That is already how such problems are solved (I'm a systems designer/engineer/programmer for 35+ years working in multivariable real-time optimization).

The issue is, at the end all the parts have to be "put back together" and evaluated as a whole and thus you still need the giant data sets.