Full native representation roundtripping in the BlenderBIM Add-on

M by Moult on 17 Sep 2020, edited 28 Jan 2021 # +13 votes

The current approach in most authoring tools is as follows:

Import, by conversion to platform (Blender, FreeCAD) specific schema
Export, by conversion from platform specific schema
Life is hard, let's do an MVD.

Nobody seems to have succeeded in this approach, as there is a huge amount of data loss in this process. The largest symptom of this is retention of the IFC representation. Whenever you see geometric losing its parametric capabilities, this is a symptom of this occurring.

A common mitigation strategy is to make the conversion to/from parametric geometry smarter. This is a lot of work. FreeCAD has gone this route, and same with the BlenderBIM Add-on to some degree (much less invested than FreeCAD - Blender meshes first, and experimentally preserves some basic shapes later). Not to mention this work will never be complete - there are combinations of primitives in IFC that will never exist in any geometry kernel (alignment curves, I-beam/SHS profiles, point clouds).

I am proposing a new approach, whereby the IFC representation model is retained in the authoring app - and the authoring app offers a fine-grained editing capability to edit parts of the shape representation tree. This retention of the IFC representation will theoretically allow 100% parametric geometry round-tripping with no data loss.

In short, this button becomes possible:

This first commit creates a proof of concept, and I have successfully demonstrated a round-trip of a medium sized file full of parametric geometry with zero parametric geometry data loss (still other data loss at the moment): https://github.com/IfcOpenShell/IfcOpenShell/commit/6cc4ce46c20cd3050070e8c63306a767bf36016b

Will keep this thread updated with the progress, as I suspect this could be a game-changer for IFC authoring.

B by brunopostle on 17 Sep 2020 #

I assume you are keeping a flag for every entity, and any that haven't been edited are written back using the same IFC structure as the original file? So any that have been edited will have blenderbim IFC quirks.

M by Moult on 17 Sep 2020, edited 17 Sep 2020 # +4 votes

@brunopostle kind of, but a bit finer grain. Essentially the IFC for the representation is stored serialised in STEP against the Blender object data.

Then, if you want to edit, you have two choices - option A is to purge the existing STEP serialisation, and start afresh as a regular Blender mesh. This is almost equivalent to a faceted brep / polygonal face set, or a triangulated face set. Option B, is you can edit the parameters of the IFC representation tree elements directly, and it'll regenerate the meshed representation so you can see the impact of your changes in the viewport. How these parameters are edited is a UX exercise - for now I have simply exposed any parameter which is a float to be a slider you can move in the mesh settings. In the future, I plan to expose it as a node tree - so you can edit it similar to how you might edit a material shader tree - with the results shown in mesh form in the viewport similar to how you might parametrically build a shape in Grasshopper / Sverchok / Dynamo. This will allow you to replace portions of the tree, for example - an extruded area solid: you might want to keep the extrusion, but simply change the profile.

Each edit of a portion of the representation tree (whether you simply tweak a parameter, or even replace portions of the tree), results in a re-serialisation of the representation in STEP form.

At export time, if a STEP serialisation doesn't exist, that means you've picked option A, and so it generates a mesh in IFC as is the current behaviour. If a serialisation does exist, it just spits it out verbatim - which means much faster exports, as every time you edit a mesh it effectively is doing a mini export so it's already prepared for you.

In once sense, the existence of STEP serialisation is the "flag" that you are referring to - but not the traditional "is dirty" flag.

To achieve this is a little harder than the proof of concept shows, especially due to the mixing of colour assignments in IFC, and the nuances of representation contexts. Hopefully my text description makes sense.

T by theoryshaw on 17 Sep 2020 #

ಥ_ಥ

R by ReD_CoDE on 17 Sep 2020 # +1 votes

@Moult Happy see finally you've started this

Another question:

How to see which objects and classes "added", "edited", or "deleted"?

Did you think about it before when it comes to round-tripping?

J by JQL on 17 Sep 2020 #

Isn't this supposed to be impossible? Or would it only be impossible because software companies don't want it and only an Opensource attempt could work? Or is it that Autodesk doesn't want it and all other companies would have a lot to gain with this move?

B by brunopostle on 17 Sep 2020 #

Having fast saves and non-destructive round tripping raises IFC to a first class file format. This would allow you to work with the IFC directly rather than working in a blend file that carries IFC metadata that can be exported.

R by ReD_CoDE on 17 Sep 2020, edited 17 Sep 2020 #

@brunopostle you're totally right, however, still BlenderBIM and also FreeCAD are dependent to their "geometric modeling kernels" for representing the BIM model (even if BlenderBIM or FreeCAD strictly don't let the original file/model(ing) changes)

This is why still I'm looking for a desired "geometric modeling kernel" (which mainly supports NURBS and also volumetric geometries)

M by Moult on 17 Sep 2020 #

@brunopostle yep - and if this experiment works with representations, which is the most complex of them all, then we could extend the experiment to other types of metadata. In theory, it could hold the IFC in memory, so that during export it's simply a file write operation. In short, the export is as fast as it can be, because it is progressively modifying IFC as you edit.

@JQL the first commit shows that it is possible - you are free to try it out :)

@ReD_CoDE this approach decouples the geometric kernel from the IFC manipulation, so the kernel doesn't really matter too much, so long as it can visually approximate the result.

R by ReD_CoDE on 17 Sep 2020 #

@Moult in representing that makes sense, but in modeling, I'm not sure!!!

We probably want to "interact" with both visual and textual features, but it seems that you want textual feature (I mean parametric interface features) does all the task which is really hard, and is something like coding but in its easier way

However, I like the approach, because it's something like "templates"

S by stephen_l on 18 Sep 2020 #

Such approach also make it possible to spread changes through speckle, brillant !

R by ReD_CoDE on 18 Sep 2020 #

@stephen_l Dion, finally have started to focus on "geometry" because the majority of solutions have issues in "geometry round-tripping"

Speckle burn to solve this issue

However, if I'm wrong please correct me, as far as I know, Speckle follows the "push-pull" approach, which is not interesting to me

Also, the approach Dion has started becomes so close to the idea of "IfcXtreme" that I shared a year ago, but the approach I shared was/is "node-based"

But today I follow another approach, which is not just about covering IFC

S by stephen_l on 18 Sep 2020, edited 18 Sep 2020 # +1 votes

From my perspective, they now share some kind of granularity by exchanging incremental modifications only - being "real-time" push/pull or in memory does not matter.

Don't know details about speckle way to handle geometry.

Another step could be low level version control protocol integration.

R by ReD_CoDE on 18 Sep 2020 #

bSI doesn't want to go a step ahead (ISO 10303-21 v3), otherwise, the majority of current issues are solvable

Y by yorik on 20 Sep 2020 # +1 votes

this is a very interesting idea.. although, just by importing an ifc file that way, and chunking everything into pieces so each piece fits in one blender object, you'll inevitably modify the ifc data. what for ex. if two objects share a same cartesian point? you'll need to split, and on reexport the ifc data won't be the same anymore. i'm sure we don't need to search much to find more serious cases. but i'd also like to try that in freecad in a form or another, it would be a benefit anyway, be it only for speed. is there a mechanism already in ifcopenshell to gather the entire dependency tree of an entity?

M by Moult on 20 Sep 2020 # +1 votes

@yorik yeah - file.traverse() and file.get_inverse() are your friends when gathering the dependency tree.

You are right, that chunking into pieces creates big problems when trying to merge them back - e.g. when two representation items refer to the same surface style. There are ways to mitigate, but none of them are easy either. I'm rethinking the approach now - I have a new idea to simply record a STEP ID "entry point" for each rooted element, and the export simply purges non referenced non rooted elements ...

It will require a few weeks to crack properly into a prototype, and would likely require a rewrite of a large portion of the BlenderBIM Add-on code... but the pay-off is also significant!

R by ReD_CoDE on 20 Sep 2020, edited 26 Sep 2020 #

! Still, I mention that you guys are in a wrong path

! Personally I count Dion and FreeCAD team as "amateurs" who know "a little bit BIM"

! Why? First of all, I requested Dion to add query feature, BUT not on Ifc file

! For you less experienced guys, Ifc file is important, for me "IFC schema/spec"

! And I have a news, soon we can parse IFC EXPRESS schema to JSON and JS/TS and maybe "SQL and SQLite" :))

T by theoryshaw on 20 Sep 2020 # +4 votes

$("a[href$='/profile/ReD_CoDE']").parents("li").css({"display": "none"});

R by ReD_CoDE on 20 Sep 2020 #

@theoryshaw you can do that, I don't care

I just don't want Dion builds another FreeCAD

M by Moult on 28 Sep 2020, edited 28 Sep 2020 # +3 votes

An improved approach in the latest commit shows that instead of chunking, it is feasible to read from the original file (more efficient, less bloat in export). It is also feasible to preserve style information! (In the first prototype, the shape was preserved, but the colours were lost. In this one, both are preserved!)

This highlights significant shortcomings in the add-on's handling of materials, and may be something I need to focus on for the next release.

P by pixelpartner on 4 Oct 2020 # +2 votes

That is a radical and revolutionary approach, most proprietary tools won't follow IMHO. Please go for it ??

M by Moult on 1 Dec 2020, edited 1 Dec 2020 # +18 votes

As promised, in the last release, I focused heavily on improving the add-ons handling of materials. It has been further polished and bugfixed in the repository. Now that materials is basically improved drastically, I have returned to working on supporting round-tripping.

I'm really proud to announce that I've done a lot of work on this and I'm now able to somewhat confidently round-trip files :) You can now take parametric geometry, edit the styles of it, as well as some parameters, then re-export it out, and it'll intelligently modify just what it needs to modify and keep everything else the same. The roundtripping supports:

Maintaining material assignment, which is now fully decoupled from the mesh
Maintaining style assignment, which is now decoupled from the mesh definition (note: for the moment, no UI is available to add or remove style items, so you can change colours, but not add new coloured portions)
Maintaining presentation layer assignments (a.k.a CAD layers)
Maintaining all geometric representation contexts. Yes, you read that right. The BlenderBIM Add-on will import the body representation, but remember all possible alternative representation contexts. Upon export, all representations will also be exported. So you can roundtrip without loss of all contexts.
Maintaining parametric geometry across IFC versions (within reason). So long as the schema is mostly unchanged, it supports automatically upgrading or even downgrading parametric geometry to/from IFC2X3<->IFC4. Note that complex scenarios like downgrading polygonal face sets to faceted breps are not currently supported... but could theoretically be!
Maintains non-rooted element reuse from the original file. No doubling up of coordinates / vectors. If it's reused in the original, it'll be reused in the roundtrip.

Still plenty of polish and especially UIs to be built to allow user to effectively manipulate this stuff, but the foundations are there.

In the next release, would advise people interested in this feature to enable the checkbox and start testing it vigorously!

I by iosvarms on 1 Dec 2020 # +2 votes

Super Awesome . Sorry, just "awesome" is not enough for this kind of improvement. :)

M by Moult on 7 Jan 2021, edited 8 Jan 2021 # +10 votes

In the ongoing quest to finally beat the round-tripping problem, I am extending the strategy past round-tripping geometry to round-tripping everything. This will solve the data loss problem on things like ownership history (yes, the BlenderBIM Add-on will be the first to fully support real ownership histories and versioning), all the unimplemented groups, structural connections, MEP stuff, construction sequencing, costing, massive files which cannot be atomically processed, STEP ID shared data, and so on.

More details to be found here: https://github.com/IfcOpenShell/IfcOpenShell/issues/1222

I am posting because within a week or so, I expect most of the core modules to be ported, and would need testers to make sure that the next BlenderBIM Add-on release is not a huge regression. Given the immense scope of the change, I'd like to have a video screenshare with any testers about the changes so that they aren't going in blind :) Let me know if interested!

J by John on 8 Jan 2021 # +1 votes

@Moult please add me as a tester - i'm not quite sure on my competency yet, but will give it my best shot