How Python Enumerations Make Data Configuration Elegant
Overview of Your Journey
1 – Introduction
When writing Python code in machine learning projects, there is almost always a need for Configuration code. This is code that keeps track of information that is used to configure other components of your code. While this definition probably helps no one that does not already know what this means, a few examples really help:
- Hyperparameters: In machine learning, hyperparameters are used to train multiple models with slight variations. The hyperparameter might be the number of trees in a random forest. Another example is the relaxation parameter for Ridge or Lasso regression.
- Access information: When connecting to a database or a cloud-hosted storage account, there is a need for access information. This includes the name, password, and other additional properties of the storage system. Similarly, say you are subscribing to a pub/sub-system like Kafka or an MQTT broker. Then the name of the topic, a password, and port numbers are necessary configuration code.
- Pipeline information: When building a data engineering pipeline, there is a need for configuration code. This can be ratios for splitting testing and training sets, or a CRON expression for when to run the pipeline.
In summary, configuration code can in some cases help to access outside systems like databases and pub/sub-systems. In other cases, it can help to configure internal systems like machine learning models and data engineering pipelines.
It often makes sense to store configuration code in separate files (in formats like .env, .yaml, or .json). Nevertheless, this configuration code still needs to be accessed within the Python code. There are many ways to organize the configuration code in your Python scripts
