UNITED STATES
EUROPE
UNITED STATES
EUROPE
  • Contact Us
    •  
    Blogs   News   Resources  
     

    Voices | Integrating Dataiku and PyCharm for Python Development

    Learn how to set up PyCharm to work seamlessly with Dataiku, from plugin configuration and environment setup to opening remote recipes, syncing changes and running local debugging sessions.

    Modern data teams rely on flexible development workflows that allow them to move seamlessly between cloud-based platforms and local IDEs. While Dataiku provides powerful in-platform tools for writing and running Python code, many developers still prefer the speed, familiarity and extensibility of PyCharm.

    By integrating PyCharm directly with a Dataiku Design Node, teams can debug Python recipes locally, manage environments more easily and streamline complex development work - all without breaking the collaborative model Dataiku is built on.

    Collaborative data tools

    In the ever-expanding Data Science universe, the need for powerful, collaborative team tools has never been more essential. In the past 5 years, Dataiku has emerged as a market leader in Data Science orchestration and is the leader in including technical and non-technical roles in collaborative projects.

    Dataiku and Pycharm Author

    For technical users, Dataiku provides the power of Jupyter notebooks out of the box for Python code development — but for those requiring more flexibility, it’s also possible to integrate directly with the PyCharm IDE and debug on your desktop.

    Let's walk through the configuration and setup of PyCharm and a Dataiku DSS Design node as well as code modification and the execution of a debugging session.

    Note: Dataiku integration with PyCharm will not be possible if you’re using the Free (Community) edition of Dataiku, which does not allow for API integration.

    DSS project

    This Dataiku project example has a single Python recipe that we’d like to connect to with PyCharm and debug interactively. With this integration we can also edit DSS Plugins, SQL Recipes and R Recipes with PyCharm.

    Dataiku and Pycharm 10

    Install the Dataiku plugin extension

    Assuming you already have PyCharm installed (the Community Edition will work just fine for this tutorial), the first step required to integrate Dataiku is to install the Dataiku Plugin from the marketplace. Simply search for “dataiku” and then click “Install” on the Dataiku DSS plugin result.

    Dataiku and Pycharm 6

    Once you have the extension installed, you’ll want to configure it to integrate with your Dataiku Design Node instance. To do this, open the preferences panel as shown in the following screenshot and navigate to the “Dataiku DSS Settings” tab. From this screen, click the “+” to add a new DSS Instance.

    Dataiku and Pycharm 2

    The “Base URL” is simply the http(s) url to the Dataiku Design node you’d like to connect. For those unfamiliar, the API key is the “Secret” value found under the “API keys” tab in your Dataiku user profile.

    Dataiku and Pycharm 9

    It's all about the environment

    If you’ve done much Python development, you’re likely familiar with virtual coding environments. In Dataiku, we’re able to create custom Code Environments for our plugins and scripts using the Code environment tab in the Administrative panel. When we integrate with a DSS Design Node instance from PyCharm, we want to create (at least) one PyCharm project for each DSS Code Environment in order to run our code locally.

    With the DSS plugin installed, we’ll create a new project in PyCharm that will align with one of the Code Environments you in your DSS instance. For example, in my instance, I have a Python 3.6 environment with no custom PIP packages installed. To re-create this environment locally, I’m going to start a new PyCharm project and select the “New Virtual Environment using Virtualenv” option, using the same Python version that I’m using in DSS. If you are using Python 2.7 in your DSS environment for example, you’ll want to select a path to a local Python 2.7 install.

    Dataiku and Pycharm 3

    Create a PIP requirements file

    Now that we’ve created a PyCharm project with a new Virtual Environment, the next step is to create a requirements file which will define the PIP packages that are required to debug our files.

    In your project, create a new text file with the File=>New=>File menu. In this file, we’ll be putting a couple essential packages needed for this integration, along with all of the PIP packages used in the DSS code environment that is used by your remote script. Name this new file requirements.txt

    Listed below are the packages that are essential to run any local environment. Again, you’ll also want to add any additional PIP packages used by your Python recipe. Add these lines to your requirements.txt file:

    <em>dataiku-api-client
pandas
numpy==1.19.3 # </em><strong><em>for windows, MUST be 1.19.3 until bug fix in 1.19</em></strong>

    With this file created, we’re now ready to open a file from our remote DSS instance in PyCharm.

    Opening a Python Recipe from DSS

    With the DSS Plugin installed, pulling Python files from your Dataiku flow is integrated right into the PyCharm menu. Navigate to File=>Open Dataiku DSS, and if your plugin has been configured correctly, you should see windows similar to the screenshots below which allow you to navigate through your DSS project recipes and select a Python script.

    Dataiku and Pycharm 7

    Opening a Recipe from DSS instance

    When you’re selecting a file, be sure to leave the “Generate Runtime Configuration(s)” checkbox selected and also select the “Install” button as shown in the above screenshot which will install the necessary Dataiku API PIP packages into your local Virtual Environment.

    Dataiku and Pycharm 1

    Install the Dataiku Client library. With that complete, click “Finish” and your remote Python file should be opened in PyCharm!

    Debugging in PyCharm

    Now that you have your Virtual Environment set up locally and the remote file open, you’ll notice that PyCharm is prompting to install the missing requirement that we’ve defined in the requirements.txt file. ahead and click “Install requirement” to add these packages to your local environment.

    Dataiku and Pycharm 11

    There is an important piece of the puzzle in the PyCharm integration to note. If you navigate to the Run=>Edit Configurations menu, you’ll see the debugging configuration that has been generated by the Dataiku plugin.

    In this configuration, the Environment variables has been populated with the DKU_CURRENT_PROJECT_KEY key set to the name of your selected Dataiku project. This is a very nice feature, making things a bit easier than the Visual Studio Code integration - but be aware that this debug configuration is specific to this DSS project.

    Dataiku and Pycharm 8

    Start debugging

    With the Python file open, add a breakpoint somewhere in your code and click the Run => Start Debugging menu item. This will start a debugging session which have full access to your DSS datasets and, if configured correctly, will run in a fully interactive debugging session allowing you to pull data from your DSS instance!

    Dataiku and Pycharm 4

    File modifications

    Of course debugging is one of the exciting features available with this integration, but we can also make local edits to the Python file and seamlessly save them back to our Dataiku instance.

    To configure this integration, let’s refer back to the DSS Settings panel in PyCharm. In this panel, you’ll see a “Automatic synchronization” option which will determine whether the changes you make to files in PyCharm will be sent immediately to the DSS server or will require manual synchronization. If you prefer to send your file changes to the Design node manually, uncheck the “Automatic synchronization” box.

    Dataiku and Pycharm 5

    Dataiku automatic synchronization

    Once you’ve verified the synchronization setting, you should be able to modify your local version of the remote Python file. If you’ve chosen to synchronize manually, you can send your modifications to the server by selecting the File=>Synchronize with DSS menu option.

    Dataiku and Pycharm 7

    Bringing PyCharm into your Dataiku toolkit unlocks a smoother development experience, especially for teams who prefer a full-featured IDE for writing and testing Python. With the plugin configured, environments aligned and sync controls in place, you can debug and iterate locally while still benefiting from Dataiku’s orchestration and collaboration features. It’s a simple integration that delivers a meaningful boost to day-to-day development.

    Game Day Predictions Author

    About the author

    With over 20 years of experience as a Lead Software Architect, Principal Data Scientist, and technical author, Ryan Moore is the SVP of AI and Analytics at Amplifi, and our resident Dataiku Neuron. He provides Data Science architecture and implementation consultation to organisations around the globe.

    If you would like to speak with Ryan about Snowflake, Dataiku, data science or AI (in any order!), please fill out our contact form here.

    About
    Services
    Partners
    News & Blogs
    © Amplifi
    Privacy Policy | Cookie Policy | Site Maintained by Honcho