Python error environment variable

Working with Environment Variables in Python

Environment variables provide a great way to configure your Python application, eliminating the need to edit your source code when the configuration changes. Common configuration items that are often passed to application through environment variables are third-party API keys, network ports, database servers, and any custom options that your application may need to work properly.

In this article I’m going to share some of the techniques and tools available in Python to work with environment variables.

How to access environment variables from Python

Using the os.environ dictionary

In Python, the os.environ dictionary contains all the environment variables. The simplest way to retrieve a variable from inside your application is to use the standard dictionary syntax. For example, this is how you can access an environment variable named USER :

Using this method, if you try to import an environment variable that doesn’t exist Python will raise a KeyError exception:

Using os.environ.get()

Getting the KeyError is a good idea for environment variables that your program requires, but there are situations where you may want to allow some variables to be optional. To avoid the error you can use the dictionary’s get() method, which returns None when the requested key does not exist in the dictionary:

Adding a default value if the variable is not defined

If you’d like to provide a default value for the missing variable that is not None , you can add it as a second argument:

Using the os.getenv() function

Python also provides the os.getenv() function to access environment variables. This function works in a very similar way to the os.environ.get() method. Here is how to access a variable with it:

This function does not raise an error for missing variables, it returns None just like os.environ.get() . And it also accepts a second argument with a custom default value:

Is os.getenv() better than os.environ ? That is really up to you. I personally prefer to use the os.environ dictionary, since it gives me the option of halting the program with a KeyError if a required variable is missing.

How to set environment variables

In this section I’m going to give you a quick summary of how to set environment variables in a terminal or command prompt window. If you want to know all the possible ways to set environment variables, my colleague Dominik Kundel has written a very detailed blog post on this subject titled How to Set Environment Variables.

Unix and MacOS

There are two basic ways to set an environment variable from a bash or zsh terminal session. One is using the export keyword:

A variable that is set in this way will be passed on to any programs or scripts that you start from this terminal session. Keep in mind that environment variables are not saved anywhere outside of the context of the shell session, so they will go away when you close the terminal session.

An alternative way to define an environment variable is to set it in the same line where the target application is executed:

This second form has the benefit that the variable is only set in the environment space of the intended application.

Microsoft Windows

If you are using Windows you have a few options. If you are interested in setting environment variables via the control panel, see the article linked above.

If you are in a command prompt window, you can set an environment variable using the set command:

Like in the Unix case, the variable is not stored or remembered beyond the current session.

If you are using the newer PowerShell console, the syntax for setting environment variables is completely different:

Finally, if you are using a Unix compatibility layer such as WSL or Cygwin, then you must go to the Unix section above and use any of the methods listed there for bash or zsh.

Using .env files

Are you confused by all the different ways to set environment variables? I personally find it inconvenient that each platform or shell requires a different procedure.

In my opinion, a better way to manage your environment variables is to store them in a .env (pronounced dot env) file. A .env file is a text file in which the variables are defined, one per line. The format of a .env file is exactly the same under all operating systems, so .env files make working with environment variables uniform across all platforms. And as if this isn’t enough, having your environment variables written in a file that is automatically imported by Python means that you don’t have to manually set them every time you start a new shell.

Here is a short .env file example with two variables:

You can create a .env file in the root directory of each of your projects, and that way you can keep all the variables that are needed by each project neatly organized!

The python-dotenv package allows a Python application to import variables defined in a .env file into the environment. You can install python-dotenv in your virtual environment using pip :

Below you can see how to import a .env file into a Python application:

The load_dotenv() function will look for a file named .env in the current directory and will add all the variable definitions in it to the os.environ dictionary. If a .env file is not found in the current directory, then the parent directory is searched for it. The search keeps going up the directory hierarchy until a .env file is found or the top-level directory is reached.

If you want to prevent python-dotenv from searching for a .env file through your directories, you can pass an explicit path to your file as an argument to load_dotenv() :

There are some additional arguments that you can use when you call the load_dotenv() function. If you want to learn about them consult the documentation.

Once the .env file is imported, you can access environment variables using any of the methods shown above.

A note about .env file security

In many cases you will be adding environment variables that contain sensitive information to your .env files, like passwords or API keys. For that reason, in general you do not want to add these files to your project’s source control repository.

The standard practice is to add an exception for files with this name, so that they are not committed to source control by mistake. For git, you can add a line with the name of the file to the .gitignore file in the root of your repository.

But if you cannot commit the .env file, how do you tell users of the project which variables need to be set? For this you can add an example .env file to your repository with a name such as .env.example, that contains the list of variables that the project requires, but without including their values. This serves as guidance for users of your project, without giving away sensitive information.

Conclusion

I hope this article was useful to help you understand how to use environment variables to configure your Python projects.

Do you have any other ways to work with environment variables? I’d love to know!

Miguel Grinberg is a Python Developer for Technical Content at Twilio. Reach out to him at mgrinberg [at] twilio [dot] com if you have a cool Python project you’d like to share on this blog!

Источник

Python Configuration Error: ‘PYTHON_BIN_PATH’ environment variable is not set #9436

Comments

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): N/A
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Mac OS X Sierra
• TensorFlow installed from (source or binary): N/A (compiling from HEAD)
• TensorFlow version (use command below): N/A (see above)
• Bazel version (if compiling from source): 0.4.5-homebrew
• CUDA/cuDNN version: none (AMD GPU)
• GPU model and memory: Radeon Pro 460
• Exact command to reproduce:

sudo bazel build —config opt —copt=-msse4.1 —copt=-msse4.1 —copt=-mavx —copt=-mavx2 —copt=-mfma //tensorflow/tools/pip_package:build_pip_package

Describe the problem

Trying to build tensorflow from source (version installed via pip does not use some optimised CPU instructions), get the following error:

Note that I used which python3 to find where my Python binary is located, and exported that as PYTHON_BIN_PATH, but it does not seem to help.

The text was updated successfully, but these errors were encountered:

Also (unrelated), the tools/tf_env_collect.sh script does not work::

Did you run configure ?

@girving I had the same error when I’m build tensorflow-serving.And I’ve configured the tensorflow alreay.

@caisq Is tf_env_collect supposed to work on Mac? It appears Linux only since it has an unguarded reference to /proc .

cc @aselle (author) for whether tf_env_collect.sh is supposed to work on Mac.

I am getting the same error trying to run command ‘bazel build —config opt tensorflow/examples/image_retraining:retrain’ Error: ‘PYTHON_BIN_PATH’ environment variable is not set and referenced by ‘//util/python:python_headers’.

I think this is a recent problem with the current master’s source codes. I tested the compilation of Tensorflow serving both locally and in docker. Both environments gave me the same error «Python Configuration Error: ‘PYTHON_BIN_PATH’ environment variable is not set». Of course, I set the configure before the build. In fact, I was able to successfully compile Tensorflow under the same environment with the same configuration a few weeks ago and there was no problem. So this must be a recent issue.

The full error message is:
`ERROR: /root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/util/python/BUILD:5:1: no such package ‘@local_config_python//’: Traceback (most recent call last):
File «/root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/third_party/py/python_configure.bzl», line 180
_create_python_repository(repository_ctx)
File «/root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/third_party/py/python_configure.bzl», line 157, in _create_python_repository
_get_env_var(repository_ctx, _PYTHON_BIN_PATH)
File «/root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/third_party/py/python_configure.bzl», line 48, in _get_env_var
_python_configure_fail(«‘%s’ environment variable is no. )
File «/root/.cache/bazel/_bazel_root/f8d1071c69ea316497c31e40fe01608c/external/org_tensorflow/third_party/py/python_configure.bzl», line 36, in _python_configure_fail
fail(»
%sPython Configuration Error:%. ))

Python Configuration Error: ‘PYTHON_BIN_PATH’ environment variable is not set`

Источник

Using Environment Variables in Python for App Configuration and Secrets

As a developer, you’ve likely used environment variables in the command line or shell scripts, but have you used them as a way of configuring your Python applications?

This guide will show you all the code necessary for getting, setting, and loading environment variables in Python, including how to use them for supplying application config and secrets.

Why use environment variables for configuring Python applications?

Before digging into how to use environment variables in Python, it’s important to understand why they’re arguably the best way to configure applications. The main benefits are:

• Deploy your application in any environment without code changes
• Ensures secrets such as API keys are not leaked into source code

Environment variables have the additional benefit of abstracting from your application how config and secrets are supplied.

Finally, environment variables enable your application to run anywhere, whether it’s for local development on macOS, a container in a Kubernetes Pod, or platforms such as Heroku or Vercel.

Here are some examples of using environment variables to configure a Python script or application:

• Set FLASK_ENV environment variable to «development» to enable debug mode for a Flask application
• Provide the STRIPE_API_KEY environment variable for an Ecommerce site
• Supply the DISCORD_TOKEN environment variable to a Discord bot app so it can join a server
• Set environment specific database variables such as DB_USER and DB_PASSWORD so database credentials are not hard-coded

How are environment variables in Python populated?

When a Python process is created, the available environment variables populate the os.environ object which acts like a Python dictionary. This means that:

• Any environment variable modifications made after the Python process was created will not be reflected in the Python process.
• Any environment variable changes made in Python do not affect environment variables in the parent process.

Now that you know how environment variables in Python are populated, let’s look at how to access them.

How to get a Python environment variable

Environment variables in Python are accessed using the os.environ object.

The os.environ object seems like a dictionary but is different as values may only be strings, plus it’s not serializable to JSON.

You’ve got a few options when it comes to referencing the os.environ object:

I personally prefer version 3 as it’s more succinct, but will stick to using os.environ for this article.

Accessing a specific environment variable in Python can be done in one of three ways, depending upon what should happen if an environment variable does not exist.

Let’s explore with some examples.

Option 1: Required with no default value

If your app should crash when an environment variable is not set, then access it directly:

For example, an application should fail to start if a required environment variable is not set, and a default value can’t be provided, e.g. a database password.

If instead of the default KeyError exception being raised (which doesn’t communicate why your app failed to start), you could capture the exception and print out a helpful message:

Option 2: Required with default value

You can have a default value returned if an environment variable doesn’t exist by using the os.environ.get method and supplying the default value as the second parameter:

If the variable doesn’t exist and you use os.environ.get without a default value, None is returned

Option 3: Conditional logic if value exists

You may need to check if an environment variable exists, but don’t necessarily care about its value. For example, your application can be put in a «Debug mode» if the DEBUG environment variable is set.

You can check for just the existence of an environment variable:

Or check to see it matches a specific value:

How to set a Python environment variable

Setting an environment variable in Python is the same as setting a key on a dictionary:

What makes os.environ different to a standard dictionary, is that only string values are allowed:

In most cases, your application will only need to get environment variables, but there are use cases for setting them as well.

For example, constructing a DB_URL environment variable on application start-up using DB_HOST , DB_PORT , DB_USER , DB_PASSWORD , and DB_NAME environment variables:

Another example is setting a variable to a default value based on the value of another variable:

How to delete a Python environment variable

If you need to delete a Python environment variable, use the os.environ.pop function:

To extend our DB_URL example above, you may want to delete the other DB_ prefixed fields to ensure the only way the app can connect to the database is via DB_URL :

Another example is deleting an environment variable once it is no longer needed:

How to list Python environment variables

To view all environment variables:

The output of this command is difficult to read though because it’s printed as one huge dictionary.

A better way, is to create a convenience function that converts os.environ to an actual dictionary so we can serialize it to JSON for pretty-printing:

Why default values for environment variables should be avoided

You might be surprised to learn it’s best to avoid providing default values as much as possible. Why?

Default values can make debugging a misconfigured application more difficult, as the final config values will likely be a combination of hard-coded default values and environment variables.

Relying purely on environment variables (or as much as possible) means you have a single source of truth for how your application was configured, making troubleshooting easier.

Using a .env file for Python environment variables

As an application grows in size and complexity, so does the number of environment variables.

Many projects experience growing pains when using environment variables for app config and secrets because there is no clear and consistent strategy for how to manage them, particularly when deploying to multiple environments.

A simple (but not easily scalable) solution is to use a .env file to contain all of the variables for a specific environment.

Then you would use a Python library such as python-dotenv to parse the .env file and populate the os.environ object.

To follow along, create and activate a new virtual environment, then install the python-dotenv library:

Now save the below to a file named .env (note how it’s the same syntax for setting a variable in the shell):

Then save the following to dotenv-test.py :

Then run dotenv-test.py to test the environment variables are being populated:

While .env files are simple and easy to work with at the beginning, they also cause a new set of problems such as:

• How to keep .env files in-sync for every developer in their local environment?
• If there is an outage due to misconfiguration, accessing the container or VM directly in order to view the contents of the .env may be required for troubleshooting.
• How do you generate a .env file for a CI/CD job such as GitHub Actions without committing the .env file to the repository?
• If a mix of environment variables and a .env file is used, the only way to determine the final configuration values could be by introspecting the application.
• Onboarding a developer by sharing an unencrypted .env file with potentially sensitive data in a chat application such as Slack could pose security issues.

These are just some of the reasons why we recommend moving away from .env files and using something like Doppler instead.

Doppler provides an access-controlled dashboard to manage environment variables for every environment with an easy-to-use CLI for accessing config and secrets that work for every language, framework, and platform.

Centralize application config using a Python data structure

Creating a config specific data structure abstracts away how the config values are set, what fields have default values (if any), and provides a single interface for accessing config values instead of os.environ being littered throughout your codebase.

Below is a reasonably full-featured solution that supports:

• Required fields
• Optional fields with defaults
• Type checking and typecasting

To try it out, save this code to config.py :

The Config object exposed in config.py is then used by app.py below:

Make sure you have the .env file still saved from earlier, then run:

You can view this code on GitHub and if you’re after a more full-featured typesafe config solution, then check out the excellent Pydantic library.

Summary

Awesome work! Now you know how to use environment variables in Python for application config and secrets.

Although we’re a bit biased, we encourage you to try using Doppler as the source of truth for your application environment variables, and it’s free to get started with our Community plan (unlimited projects and secrets).

We also have a tutorial for building a random Mandalorion GIF generator in Python that puts into practice the techniques shown in this article.

Hope you enjoyed the post and if you have any questions or feedback, we’d love to chat with you over in our Community forum.

Big thanks to Stevoisiak, Olivier Pilotte, Jacob Kasner, and Alex Hall for their input and review!

If you’re new to Python, check out Cory Althoff’s Python environment variables primer which covers entry-level concepts in more detail.

Источник