Decision Optimization

I'd like to tell you about a new cool feature of Modeling Assistant (MA) available in IBM Decision Optimization. It's an advanced feature of MA which lets you add any constraint, beyond the predefined templates provided for the MA domains, to your model.

This article was co-authored with Hugues Juille and Xavier Ceugniet

Modeling Assistant

Before getting into the details of the new feature, you need to know about the Modeling Assistant. There is a great introduction to MA: Decision Optimization Modeling Assistant. There's also a tutorial: Formulating and running a model: house construction scheduling

TL;DR: Modeling Assistant allows you to solve quantifiable business problems using natural language rules without sophisticated Operations Research expertise or even programming knowledge as is typically needed for such problems.

Figure above: from input data via Modeling Assistant to the solution on a Gantt chart.

Limitations of MA

For cases that are fully covered with the implemented domains and rules, Modeling Assistant works great. You can select a domain and setup a basic model with a few clicks. Then use the natural language search field to find additional rules to be added to the model, complete the rules with parameters, adjust weights, all with a graphical UI. The problem starts when there is no predefined template for a rule that would cover a constraint that should be added to the model. You just cannot select an option if it isn't there.

Custom constraints to the rescue!

Now with the new feature you could extend an MA model with custom constraints that allow you to write a custom Python code using the DOcplex API that actually implements the very specific constraint you need.

Note: this custom code approach is supported for constraints only. Custom objectives are not available yet.

Collaboration flow

OK, so you hear me saying the power of MA is that you can work with OR models without OR or programming expertise.. and that you can write some custom DOcplex code using Python to address some particular constraint? Now you wonder.. does that make sense? I mean, if someone uses MA that's because they don't want to, or even don't know how to, code their model in Python, isn't it? So.. what's the point?

The key to answer this contradiction is that there could be more than one persona involved. For example:

There is a business user, who is using MA for scheduling problem. They don't want to learn Python, CPLEX or constraint programming. They don't really need, because MA allows them to solve scheduling problems with IBM Decision Optimization using easy UI where constraints expressed in plain English can be managed. That works well until there is a new business restriction that cannot be modelled using the existing predefined rules of MA. This could be enforced externally, e.g. by a new provision of labor law, or internally, or by some business policy or limitation. So far, the business user would be stuck at this point, being limited to the set of predefined rules that were implemented in the Modeling Assistant.
The only workaround up until now was to export the model to a notebook and ask someone skilled in Operations Research and Python to implement the missing constraints. There was a big issue with this workaround: once exported to a notebook, there was no way to convert it back to the MA, so the business user could not work on the model any further.
Now the business user can ask for help an Operations Research (OR) expert and keep control of the model afterwards. After gathering requirements from the business user, the OR expert could add a custom constraint in the MA model. They can both agree on the specification of a rule, i.e. how the new rule should be verbalized and what parameters could be provided. Then the OR expert implements a Python function that adds relevant constraints using the DOcplex API. The business user can use the model further, experiment with it, changing constraints, goals and weights, modifying parameters as usual.

Defining custom constraints

To add a custom constraint to your existing Modeling Assistant model you need to:
1. Add a new custom constraint to the model - use search or browse the rules catalog. So far this is just like adding any of the predefined rules in Modeling Assistant:

2. Define details of the constraint by entering the so called Rule specification. The specification is typically just a phrase that will be used to display the constraint in the model and how it can be customized with parameters.

3. Provide or clarify parameter values if needed.
4. Edit code to enter actual Python custom code for the constraint using DOcplex API

For the exact and up-to-date step by step instructions to define a sample custom constraint, please refer to the dedicated article in the official IBM Cloud Pak for Data documentation.

Rule specification

Every goal and constraint needs to be displayed in the MA model. The text that is used to display a constraint or goal is called its verbalization. The predefined constraint templates that are available in MA domains already have meaningful verbalizations which describe it with plain English and references to your data model. Verbalizations allow a business user to understand an MA model just by looking at it and reading the goals and constraints.
Predefined constraint templates already come with appropriate verbalizations. For a user-defined custom constraint MA is unable to infer a reasonable verbalization, so it has to be explicitly provided in the first step of completing a custom constraint. Just put a sentence that describes the new constraint well. Parameters are enclosed in square brackets, i.e. [ ]. Any text outside of square brackets does not need to follow any further syntax and will be displayed literally.

Parameters

If you ever used MA before, you are already familiar with parameters. Look at the decision rule above. This is an example of a constraint template instance that has parameters. These pieces of the constraint instance verbalization are displayed in blue, are clickable, and can be changed. When you add a constraint instance from domain templates some parameters are automatically bound with an inferred value. The inference is the secret sauce of Modeling Assistant. When a parameter value cannot be reliably inferred they are displayed underlined in yellow. The user has to provide a value to complete the decision rule, otherwise the MA model is not ready to solve. Some parameters provide context to the decision rule (pointing to a concept within the data model) or exact criteria (e.g. "is less than" "5"). The role of parameters in custom constraints is similar. Some could provide context (reference to data model concepts) or exact criteria (numeric or string literals).

Parameters for custom constraints are defined within square brackets in the rule specification. When editing a rule specification, put in the brackets a phrase that closely describes the intended binding. The value could then be inferred. For example, if you enter "Assign all [subcontractors]" as the rule specification, the following would happen:

- The specification is automatically converted to canonical form with a parameter name and domain concept: "Assign all [parameter1: <Unary Resource>]". Note that the expression for parameter has been expanded. Now the parameter has a name ("parameter1") which is just a default name and an expected domain concept ("<Unary Resource>") inferred from the input for parameter. You don't need to worry about understanding the domain concepts that appear there. These are kind of generic types defined in

the selected domain, but not specific to the actual data model used in the scenario. If you are curious, go to the Data Schema tab to see the domain concepts represented by your data.

Advanced hint: This means, that you can, for example, copy a canonical rule specification from acustom constraint of a Scheduling model scenario to some other Scheduling scenario (perhaps even with different input data schema).

You can also edit the rule specification to change "parameter1" to any other name that would be more descriptive. Note that the new parameter name has to be a valid Python identifier, as this will be the parameter name in the Python function implementing the constraint.
- The value of the parameter is automatically bound to the "subcontractors" concept. You could now change the parameter (if there are more concepts of the same type in the data model) or add a filter to pass to the function only the subcontractors that meet some criteria.

Custom code in Python

After configuring the specification and parameters of the new custom constraint it is time to provide an actual implementation. This is the borderline beyond which CPLEX and Python skills can no longer be ignored. Most MA users will at this point probably ask for help from an Operations Research expert on their team who is already familiar with these technologies.

First you need to choose a function name for the implementation. Any valid Python identifier should be fine.

Then you can click the Edit Python button to provide the function implementation. It will open a simple editor with a Python class named CustomCode. The class contains several function stubs that are called at various phases of model generation. I'm not going to cover the entire lifecycle now. For the purpose of this custom constraint there should be a single function with the name you chose just a moment ago.

Such a function has at least two arguments: self and mdl. The argument mdl is an instance of the docplex.mp.model.Model or docplex.cp.model.CpoModel, depending on the domain of the model to which the custom constraint has been added. There will be additional arguments if there were any parameters defined in the rule specification. The schema of parameters that refer to data model are documented in the comment above the function.

Note: only the function body should be edited. Changes to the function signature or other changes in the CustomCode class and beyond can break the model generation and are not supported. If function name has to be changed, you need to close the editor, change the function name in the custom constraint details and open the editor again. A function stub with the new name will be there. The old function may be still there for reference. When any parameters are changed, added or removed in the custom constraint specification the Python functions are updated in the same way.

Hint: the Python editor is rather basic, so it may be more convenient to write and debug the Python code for the custom constraint in some IDE and then just copy and paste it into the editor dialog. For example, the business user exports the model to a Jupyther Notebook using the generate notebook option in the scenario. They share the notebook with the OR expert, who then find the function stub for the custom constraint in the notebook, implements and tests the function within the Jupyther environment. When the implementation is ready they can copy the function and send it to the business user who pastes it in the custom constraint code editor.

Example: add a custom constraint to a scheduling model

Now I'm going to show you an actual example based on the House construction scheduling sample. You can find the sample and instructions for loading it here: House construction scheduling sample The sample uses the Scheduling domain to build optimal schedule of house construction activities. Each of the activities is also assigned to a particular subcontractor, with respect of their skills in particular activities.

After loading you can review the rules in this model, modify constraints, remove or add new ones. To see the results Run the model. It will take a few moments and then you can see the solution - either in a tabular way on Explore solution tab or as a Gantt chart on the Visualization tab.

Now let's suppose you have some unusual requirement to take into account. For example, you don't want Jim and Jack to ever be assigned a task on the same day. Maybe they are critical in some other area of your business and at least one of them should always be left available there. Or maybe they just don't like each other and you know that when they meet it's just an unproductive mess. Whatever the reason, let's create a custom constraint to enforce that new requirement!

After adding a new custom rule we start with providing the rule specification. This could be something like: "Do not allow [subcontractor] and [subcontractor] work together on the same day"

You can now edit the specification to customize the parameter names. Let's rename them to "subcontractor1" and "subcontractor2".
Note that the two parameters are automatically bound to Subcontractors. This would pass data frames with the entire Subcontractor table to the function. You want to restrict particular subcontractors, so that's not yet exactly what you need. You can now click on the first blue Subcontractors parameter and type "Jim" instead. Change the other to "Jack". You can modify these parameters later if the conditions change.
Now the Python function will just get single row data frames as arguments, with Jim and Jack indexes respectively. Talking about the Python function.. we still need to give it a name. Let's call it dont_work_together.

Now you can click the Edit Python button to provide an implementation. A dialog shows up with the code editor:

Find the dont_work_together function in the opened editor and paste the following implementation there:

    def dont_work_together(self, mdl, subcontractor1, subcontractor2):
        global helper_add_labeled_cplex_constraint, helper_get_index_names_for_type, helper_get_column_name_for_property, list_of_SchedulingAssignment


        intervals_by_subcontractor = list_of_SchedulingAssignment.groupby(level='id_of_Subcontractor')['interval'].agg(list)
        
        group_1 = []
        for subcontractor in subcontractor1.index:
            group_1 += intervals_by_subcontractor.loc[subcontractor]
        group_2 = []
        for subcontractor in subcontractor2.index:
            group_2 += intervals_by_subcontractor.loc[subcontractor]
            
        state_func = state_function(name='dont work together 1')
        
        for itv in group_1:
            mdl.add(always_equal(state_func, itv, 1))
        for itv in group_2:
            mdl.add(always_equal(state_func, itv, 2))
        pass

Now close the editor, run solve and explore the solution on the Visualization tab:

The schedule is slightly longer, but Jim and Jack don't meet together anymore. Well done!

Summary

You've seen the new capability of custom constraints in Modeling Assistant. You've learned what are they, what their purpose is and how to use them.
This is not a feature that every user of the Modeling Assistant will use or even need to use. For sure it is not as easy to use as the predefined constraint templates.

However, I believe there are cases where it will make a big difference between being stuck or covering that last missing business requirement that makes the solutions provided by the MA relevant.
I hope you'll find it useful!