Optimising optical systems means systematically searching for the parameter values that best satisfy a given criterion or set of criteria. This can be lens geometry, element positions, radius of curvature, refractive index, layer thickness, material parameters, wavelength — or, at the integrated-photonics level, the parameters of waveguides, couplers, or interferometers. This article is a natural continuation of the component → PIC → system series: after simulating an optical link comes the question of how to choose the parameters to achieve the required result.
First the question, then the optimisation
One of the most common mistakes is starting with „how do I optimise this system?” without first determining „what exactly do we want to optimise?”. There is no single universal definition of the best optical system: for one project the priority is maximum power at the detector, for another minimum aberration, in a transmission system minimising BER, in a WDM system the spectral response and channel separation, in a PIC low loss, a specific phase response, or coupling between ports.
That is why the first step is defining the objective function — for example: maximise transmission; minimise optical loss; minimise the difference between the designed and the achieved response; minimise aberration; maximise the bandwidth while keeping loss below a given level. Without this, the optimisation algorithm has no unambiguous task.
The objective function — the heart of optimisation
The objective function (also called the cost function or merit function, depending on the convention) describes how good a given design variant is. If we design an element meant to maximise transmission, we can define F = T and look for the maximum; if we want to minimise loss — F = loss and look for the minimum.
In a realistic project the objective function is often complex — we want high transmission, a wide band, a limited element size, and no violation of the technology parameters, all at once. The problem becomes multi-objective. You can use a combination of criteria, e.g. F = w₁·loss + w₂·penalty + w₃·size, where the coefficients w₁, w₂, w₃ set the importance of each term. This is not the only method, though: in many problems it is better to treat some requirements as constraints rather than terms of the objective function. Not everything that is a design requirement has to be an objective function.
The objective and the constraints are two different things
For a lens design we want to minimise aberration — that is the objective. At the same time the diameter may not exceed 20 mm, the thickness has to be less than 5 mm, and the radius of curvature has to be within a given range — those are constraints. The algorithm should look for the best solution within the feasible region. The same applies to integrated photonics: we can minimise coupler loss, but the waveguide width, device length, and technology spacings have to stay within a manufacturable range.
A well-defined optimisation problem has three elements:
- design variables — what can we change?
- objective function — what do we want to improve?
- constraints — what must not be exceeded?
- design variables — what can we change?
- objective function — what do we want to improve?
- constraints — what must not be exceeded?
Do not optimise parameters that do not matter
If a model has 30 parameters, that does not mean all of them should be optimisation variables. If one parameter has practically no effect on the result, leaving it in the process only enlarges the search space. That is why, before the actual optimisation, it is worth running a sensitivity analysis: change a single parameter over a given range and observe the change in the objective function.
If a small change in a parameter causes a large change in the result — that is a sensitive parameter. If the result barely changes — the parameter may be of little importance for this problem and can be fixed. Fewer variables usually means a simpler optimisation problem.
Optimisation does not replace understanding the physics
An algorithm may find a solution the designer did not expect — this can be an advantage, but it can also be a problem. If we do not understand why the solution works, it is hard to judge whether it is physically sensible and whether it will be robust to real tolerances. Good practice therefore looks more like: physics → model → analysis → restricting the space → optimisation → validation — rather than: model → „run optimisation” → accept the result. The algorithm searches the solution space according to a criterion; it is the designer who decides whether the solution found is actually useful.
Start with a simple model
It is tempting to build a very detailed model right away — often a bad idea. If the model contains many elements, many materials, a wide wavelength range, a large number of parameters, and detailed physical models, each single evaluation can be computationally expensive, and the algorithm may need very many such evaluations.
That is why it is worth using multi-level modelling: first a simpler model (does the solution work at all?), then a more detailed one (does it work with realistic parameters?), finally an accurate model (does it meet the requirements and is it robust to tolerances?). This is the same logic that appears when moving from a component to a PIC and on to the whole system.
Optimise at the right level
Not every problem should be solved at the whole-system level. If the problem is lens geometry — optimise the lens geometry. If the problem is coupling between waveguides — focus on the coupler parameters. If the problem is the BER of the whole link — only then should the objective function be defined at the system level.
This leads to a hierarchical strategy:
- component level — we optimise the parameters of a single element (tools such as OSLO, TracePro, or OptiFDTD have built-in optimisers — see How to choose optical simulation software?);
- PIC level — we optimise how many elements of an integrated circuit work together;
- system level — we optimise the parameters of the whole path against the final performance function.
This way a very expensive whole-system simulation is not used to solve a problem that could have been solved much earlier.
- component level — we optimise the parameters of a single element (tools such as OSLO, TracePro, or OptiFDTD have built-in optimisers — see How to choose optical simulation software?);
- PIC level — we optimise how many elements of an integrated circuit work together;
- system level — we optimise the parameters of the whole path against the final performance function.
This way a very expensive whole-system simulation is not used to solve a problem that could have been solved much earlier.
The parameter sweep is an underrated tool
Before running an advanced algorithm, it is often worth doing something simple: a parameter sweep — systematically recomputing the model for different values of a parameter (e.g. radius of curvature 10 → 12 → 14 → 16 → 18 mm) and computing the objective function at each point. You get the dependency parameter → result.
A sweep can show where the maximum or minimum is, whether there is a single optimum, whether the problem has many local extrema, and how wide the region of good solutions is. It also helps find a reasonable starting range for a more advanced optimisation.
Local or global optimiser?
There is no single best algorithm for all problems. Local methods work well when we are near a good solution and the objective function has a suitable structure. Global methods are useful when the solution space is complex and there is a risk of getting stuck in a local optimum. In practice a common strategy is: global search for a promising region → local refinement of the solution in that region.
Do not trust a single run
If an algorithm found a solution after one run, that does not yet mean it is the best possible solution — in nonlinear problems the objective function may have many local extrema. So it is worth changing the starting point, doing multiple runs, comparing the solutions, and checking the stability of the result. If different runs lead to a similar solution — confidence in the result grows. If they lead to very different solutions of similar quality — that is also valuable information: the problem may have many practically equivalent solutions.
The best result is not always the best design
Suppose we found two solutions: design A — 0.10 dB loss, design B — 0.12 dB loss. At first glance A is better. But if A requires very precise geometry, is very sensitive to temperature, has a very small technology margin, and is hard to manufacture, while B is much more robust — then B may be the better engineering design. That is why, after optimisation, a tolerance analysis has to be done.
Tolerance is part of optimisation
A real element never has exactly the parameters from the model: production has tolerances, the material may have a slightly different refractive index, the temperature changes, the element position deviates from nominal. It is therefore not enough to find the optimum x = x₀ — you have to check what happens when x is a little larger or smaller. If a small change in a parameter causes a sharp degradation of the result, the solution may be hard to use; if the result stays good over a wide range, we have a larger design margin.
Hence the conclusion: it is worth looking not only at the value of the optimum, but at the shape of the objective function near it. A flat optimum is good news — even if production does not hit the nominal value exactly, the result stays good. Such a design is often more valuable than a solution with a slightly better but very sharp minimum. This is another reason why a simple parameter sweep is so useful.
Do not optimise a single operating point
An optical system often has to operate over a range of conditions: many wavelengths, different temperatures, angles, powers, and polarisation states. If we optimise the system only for one wavelength, the response may degrade sharply outside it. That is why the objective function can be defined for the whole operating range — e.g. instead of minimising loss only at λ₀, we minimise the average or worst-case loss in the range λ₁ ≤ λ ≤ λ₂. The problem changes from „find the best point” to „find a design that works well across the whole required range”.
The average value is not always enough. If loss is 0.1 dB at nine of ten operating points and 1 dB at one, the average may look good — but that one point can be critical. When the system has a hard worst-case requirement, it is more appropriate to optimise the maximum error rather than the average.
Multi-objective optimisation and the Pareto front
In real projects we rarely have one goal — we want low loss, wide bandwidth, small size, high tolerance, and manufacturing simplicity all at once. It is not always possible to improve every parameter at the same time; there is often a trade-off: smaller device ↔ higher loss, wider band ↔ lower selectivity, maximum efficiency ↔ lower technology tolerance.
In such cases there is not always a single „best” solution. Instead you get a set of compromise solutions — the Pareto front — in which you cannot improve one criterion without worsening another. It is the designer who then picks the solution suitable for the specific application.
System-level optimisation: do not lose the final goal
At the system level we can optimise the parameters of the transmitter, channel, and receiver against BER, but at the same time we have to control power, OSNR, bandwidth, dispersion, and receiver parameters. The objective function can refer to BER but with constraints: minimise BER subject to OSNR ≥ a required value, output power ≤ a limit, and component parameters within the technology ranges.
This shows why system-level optimisation is a natural continuation of simulating an optical link: first we build the model, then we define the quality metric, then we search for the parameters that improve that quality.
Automation matters more than a „clever algorithm”
In large projects, optimisation may require hundreds or thousands of model runs. That is why automating the workflow is as important as the algorithm itself. A well-prepared process should: set parameters → run the simulation → collect the results → compute the objective function → check the constraints → pass the result to the optimiser → save the best solutions. The simulation then becomes part of a computational process rather than a manual experiment. This is especially important in environments such as OptiSystem, where component parameters, analyses, and results can be used in automated design processes.
Computation cost and surrogate models
If a single simulation takes a second, 10,000 runs may be acceptable; if it takes 10 minutes, the same process becomes a serious problem. Before running an optimisation, it is worth estimating the number of evaluations × the time of a single simulation. If the result is too expensive, you can simplify the model, reduce the number of variables, narrow the search space, use a surrogate model, or run a multi-stage optimisation.
A surrogate model is built by first generating a set of results for selected points in the parameter space and then approximating the dependency parameter → result. Such a model is much faster than a full simulation — you can run a broad search on it and verify the best solutions on the accurate physical model. A surrogate model is, however, only an approximation and should not be treated as an automatic replacement for the physical model without later validation.
First optimisation, then validation
The design process does not end the moment the algorithm returns the best set of parameters — that is only a candidate solution. Next it has to be checked on a more accurate model, over a wider parameter range, for other wavelengths, production tolerances, temperature conditions, and different operating configurations. If we optimised BER for one OSNR value, we have to check the behaviour across the whole OSNR range; if we optimised one wavelength — the whole required band. Optimisation gives a candidate; validation tells you whether the candidate is actually a good design.
A typical workflow
- Step 1 — define the requirement — what is the system to achieve?
- Step 2 — choose the metric — what will measure the quality of the solution?
- Step 3 — define the variables — which parameters can we change?
- Step 4 — define the constraints — which values are physically, technologically, or system-wise unacceptable?
- Step 5 — sensitivity analysis — which parameters really matter?
- Step 6 — a sweep — what does the solution space look like?
- Step 7 — choose a strategy — global search, local, or a combination of both?
- Step 8 — optimise — find the best candidates.
- Step 9 — check tolerances — is the solution robust to deviations?
- Step 10 — verify with an accurate model — does the result hold under realistic assumptions?
- Step 11 — check the system — does the component-level improvement actually improve the whole system?
- Step 2 — choose the metric — what will measure the quality of the solution?
- Step 3 — define the variables — which parameters can we change?
- Step 4 — define the constraints — which values are physically, technologically, or system-wise unacceptable?
- Step 5 — sensitivity analysis — which parameters really matter?
- Step 6 — a sweep — what does the solution space look like?
- Step 7 — choose a strategy — global search, local, or a combination of both?
- Step 8 — optimise — find the best candidates.
- Step 9 — check tolerances — is the solution robust to deviations?
- Step 10 — verify with an accurate model — does the result hold under realistic assumptions?
- Step 11 — check the system — does the component-level improvement actually improve the whole system?
The most common mistakes
- Optimising without a clearly defined goal — the algorithm may find a mathematically good but engineering-worthless solution.
- Too many variables — a large parameter space dramatically increases the computation cost.
- Ranges that are too wide — if parameters can take physically meaningless values, the algorithm wastes time analysing them.
- Optimising a single operating point — the solution may work great nominally but badly across the real operating range.
- No tolerance analysis — an ideal model is not the same as a real product.
- Accepting the first optimum — one run is not always enough.
- No validation — the optimiser's best result is still a model result, not a measurement.
- Optimising the wrong level — there is no point optimising the whole system if the problem is in a single component.
- Too many variables — a large parameter space dramatically increases the computation cost.
- Ranges that are too wide — if parameters can take physically meaningless values, the algorithm wastes time analysing them.
- Optimising a single operating point — the solution may work great nominally but badly across the real operating range.
- No tolerance analysis — an ideal model is not the same as a real product.
- Accepting the first optimum — one run is not always enough.
- No validation — the optimiser's best result is still a model result, not a measurement.
- Optimising the wrong level — there is no point optimising the whole system if the problem is in a single component.
Best practices in brief
- Define the problem before choosing an algorithm.
- Separate the objective function from the constraints.
- Limit the number of variables to those that really affect the result.
- Get to know the parameter space first — a sweep and sensitivity analysis are very valuable.
- Match the model level to the design question.
- Do not optimise a single point if the device has to work over a range of conditions.
- Account for tolerances already at the design stage.
- Do not equate the best mathematical result with the best engineering design.
- In expensive simulations, use hierarchical models and — where justified — surrogate models.
- Verify every optimisation result on a more accurate model and under near-real conditions.
- Separate the objective function from the constraints.
- Limit the number of variables to those that really affect the result.
- Get to know the parameter space first — a sweep and sensitivity analysis are very valuable.
- Match the model level to the design question.
- Do not optimise a single point if the device has to work over a range of conditions.
- Account for tolerances already at the design stage.
- Do not equate the best mathematical result with the best engineering design.
- In expensive simulations, use hierarchical models and — where justified — surrogate models.
- Verify every optimisation result on a more accurate model and under near-real conditions.
Optimisation as a layer at every modelling level
In this series three levels appeared: component → PIC → system. Optimisation does not create another level beside them — it is a layer operating at each of them. At the component level we optimise geometry or physical parameters, at the PIC level how many elements work together, at the system level the operation of the whole path (BER, Q-factor, OSNR, and other system parameters).
The full process can be represented as: model → simulation → analysis → optimisation → validation — and applied in turn to the component, the PIC, and the system. Good optimisation is not about finding „the best parameter”, but the best solution within the real requirements of the project. A broader view of choosing the level and method of simulation → Methods for modelling optical phenomena.
Summary
Optimising optical systems is a natural extension of simulation. Simulation answers the question „what will happen if we build a system with these parameters?”; optimisation goes a step further: „what parameters should the system have to achieve the required result?”.
Best practices do not come down to choosing a particular algorithm. The most important things are: a well-defined goal, appropriately chosen variables, realistic constraints, sensitivity analysis, the right model level, tolerances, and final validation. In photonics this is especially important, because a solution optimal for a single component need not be optimal for a PIC, and a solution optimal for a PIC need not give the best performance of the whole system. That is why the practical design workflow is: component → PIC → system, and simulation → optimisation → validation.