Section: New Results
3D cities modeling and reconstruction
WP3.1.4 Coordination and management
Works achieved in 2009 cover:
The planning and organization of research and developments works tackling this problematic:
10 progress meetings and technical workshops were organized during this period.
The supervision of Work Package delivrables:
all developments have been evaluated and associated to a delivery document beforehand their officialisation on the project's quickplace.
Technical and financial reporting to the project's coordinator and funders:
ss work package coordinator, Cedric Guiard had to synthesize achieved progress and to draw management's reports on the Work Package activies. He also had to synthesize the answers of the partners to the questions drawn by the DGCIS.
Comparison and collaboration of the reconstruction approaches:
[28] depicts the qualitative and quantitative evaluation of the three reconstruction tools developed in this Work Package. This study also explicits how the derived results can be combined through their interface with the production platform to benefit from the complementarity of these different approaches.
Final report of the Work Package:
[27] synthesizes the works and results achieved in this Work Package by the 5 partners involved in the workpackage (CMM-ENSMP, ENST THALES and INRIA). These documents list all developments, delivrables, meetings and actions realized in the context of this project as well as the load and expense invested by each partner. All documents and associated data are managed in configuration under GForge.
WP3.1.4-3: Model and Image-based reconstruction of Haussmannian's buildings
Works achieved in 2009 cover:
Definition and reduction of an architecture model through the analysis of selected instance buildings
This development concerns the definition, parametrization and serialization of a procedural model expressing and constraining the representation of buildings sharing a common architectural typology. This set of parametrizable rules, referred to as "Grammar model" implements:
-
the semantic decomposition of buildings representation into a set of construction steps
-
the relations and constrains existing among the different architectural elements
-
visual representations admissible for the terminal elements
These rules are particular to the considered architectural typology. They define a space of admissible representations for this architecture style and insure that buildings derived from this model conform with the structure and appearance of this typology. This "buildings space" must be of minimal dimensions to underlie the reconstruction process (model reduction); it must be possible to amend and potentially extend this model through the production to take into account new architectural features.
In respect with stated objectives, this work was centered on the procedural modeling of the Haussmannian typology characterized in WP3.1.4-0 . The model definition derives from the analysis and factorization of procedural rules respectively encoding a set of buildings of the considered typology. The choice of these representative buildings as well as the analysis and reduction of the model has to be supervised by a user specialized in architecture. The designed tool supports this process by enabling:
-
the dynamic definition and editing of the procedural rules characterizing the buildings instances considered to draw this analysis,
-
the interactive factorization and enrichment of these rules required to exhibit the degree of freedom and to express the constrains and relationships which regulate the parameters of this model.
A new procedural paradigm has been developed in this objective. Entitled DGA for Dependency Graph Architecture, the formalism of this shape grammar is detailed in [26] . This development also gave the possibility to dissociate the semantical decomposition of an architectural style from the structure, geometry and materials particular to buildings representations derived from this typology. This enabled to integrate a priori and to independently control the model and instances at these different levels.
This model is established through the definition of a dependency graph. This supports the interactive editing and dynamic evaluation of the model by minimizing the operators to be processed when its definition requires some modification. Global variables are defined through the nodes attributes (boolean, integer, float, ...) whose range values can be specified. Relations and constraints over the model's components are defined via the definition of arithmetics or booleans expressions among these attributes.
The dependency graph sets the decomposition and semantical characterization of the model. Nodes of this dependency graph draw the structural, geometrical and material variability of this model. The parametrization and evaluation of this dependency graph encode and generate the visual representation of buildings instances complying with this typology.
This production tool is implemented as a Maya plug-in. This enables to implicitly benefit from the functionalities and interfaces of this product. In particular,
-
the dependency nodes enable the consideration of a wide range of creation and modification operators beyond standard CGA grammar ones (Component Split, Split, Repeat, Insert, Size ...),
-
the rules are expressed through a script language which includes control over the instructions flow (conditions, loop, ...) and enable to call native Maya commands,
-
the definition, editing/reduction/enrichment and parametrization of the model can be equally handled through the DGA rules of this typology, through the dependency graph issued from the rules interpretation or at the level of the hierarchical textured mesh representation which derives from the evaluation of the model dependency graph via the various user-interfaces of Maya .
|
3D Reconstruction of geo-specific building representations
This work aimed at increasing the productivity and fidelity of the reconstruction of Haussmannian's buildings by coupling the analysis of both 2D (photography) and 3D (scan) imagery of the facade with the DGA model depicted above.
This reconstruction approach relies on the procedural definition of the geo-specific representation on different analysis components enabling to identify and to segment architectural elements within the considered acquisitions. Handled in WP3.1.4-3 , the development of this reconstruction scheme enabled to integrate the 2D/3D analysis components delivered by the implied partners INRIA, ENSMP-CMM and ENST to achieve a detailed 3D representation of the facade while retaining the editing and enrichment capabilities intrinsic to its DGA definition. This reconstruction scheme follows a coarse to fine refinement strategy based on the construction steps defined for this typology and introduces a priori on the relations and constraints existing among the elements of this architecture style.
The coupling between, in one hand the 2D and 3D data analysis and, in the other hand, the automatic or interactive parametrization of the DGA definition is done through the Scopes associated to the derived representation. A Scope defines the region in space bounding a particular architectural element (terminal or non terminal Shapes). The reconstruction scheme follows the semantic decomposition established for this typology starting with the initial facade Scope which discriminates the 2D image and 3D points data to be considered in this reconstruction. This Scope can be derived from the planimetric segmentation of the facade image (developed in WP3.1.4-5 ) and/or can be edited by the user. At each reconstruction step, a Scope allows to discriminate the 2D and 3D data regions which support the identification and segmentation of particular architectural elements. The comparison between the analysis respectively processed on the image and on the scan data enable to determine the existence of the considered architectural element (tile, balconies, apertures, ...) and to determine its relative:
-
subdivision in the respective directions or absolute dimensions
-
width and height derived from the image analysis
-
depth derived from the 3D scan analysis.
This reconstruction scheme continues down to the terminal elements of this typology. The generic representation predefined in the Shape Library are textured and 3D registered based upon the acquisitions to count for the geometric and appearance particularities of the building and to increase the fidelity of its representation. The resulting DGA characterization can be interactively adjusted or enriched at every construction steps both at the model and representation level.
The principles, developments and results of this reconstruction approach are further detailed in [29] , [28] (See also the next section).
Haussmannian Facade Analysis
Haussmannian buildings present rather regular and consistent elements that are more suitable for analysis automation. For example, these types of buildings consist of multiple highly similar floors, significant repetition of architectural elements and well defined dimensions constrained to the street width, which exhibit high degree of consistency.
Our analysis process starts with the use of single facade images, which consist of multiple colors, materials, shapes and textures, saturated with significant reflections and partially occluded by public facilities and plants. Hence in this consideration, single features will not be sufficiently the clues for image recognition and further segmentation. Consequently, we use a hybrid, multiple-clued approach, which means, the composition recipes of colors, shapes, textures will from a context for recognition of building elements, be quite similar to the interpretation of human languages. Our image-based modeling of Haussmannian facades brought three major contributions (1) An image analysis/synthesis loop for better reconstruction. (2) A joint color and edge profile-based method for building typology determination. (3) A hybrid method for architectural elements recognition.
Overview
In our approach, two steps are taken in series. Firstly the facade typology is inferred in terms of floors and tiles segmentation vertically and horizontally. Second, different architectural elements are deduced by using different image descriptors.
Determination of Facade Typology
Our assumption here is that windows always reflect or display different colors against the dark background facade walls, either the blue from the sky or various colors from surrounding objects. And this could be perceived in the hue image, which shows the global color contrast. By checking the occurrences of windows but not recognizing them, the facade typology could be determined as well as floor and tile segmentation.
As the hue information differs windows from walls, it highlights the typology of the facades and suppresses other details which may do harm to global analysis. This hue information is projected to X and Y axes to form horizontal and vertical profiles respectively which facilitate floor an tile segmentation. In this way, we reduce the 2D problem to 1D problem, which is much easier and more efficient in terms of processing time.
Detection of Architectural Elements
After determination of the facade typology in terms of floor and tile segmentation, more details will be detected to obtain a complete description of the facade. These details include windows, balconies, and various decorations. The central part of the detection of various architectural elements is window detection because windows mark the active region of residents' lives in the tile so that all other elements are positioned around those regions to functionally and decoratively highlight them.
Various elements could be detected separately according to their own image characteristics. As each element appears at multiple positions, the detection could be co-verified among multiple instances. That means if one instance shows a strange result, it could be corrected by the presence of other instances. For example, for window detection, if a few windows were badly estimated, they would be corrected by aligning them with others horizontally and vertically. Therefore, a relatively accurate detection could be achieved.
Furthermore, there is another scheme to ensure the accuracy of the detection. First, a rough detection of various architectural elements is established with various image descriptors globally. Then the detection is refined by local information. For instance, the window is detected by color difference but the final estimation of size and location are based on the full facade image.
Window Aperture Detection
The detection of windows is very difficult since the real scenarios are very complex. Windows have paneled glasses which intrinsically absorb the red rays form the light and reflects surrounding contents including the sky above or the facing objects. In addition, windows may have shutters, which have their own colors and shapes, internally or externally, open, partially close or fully close and they can be occluded by plants... Hence, windows on facades do not have a unique shape, color or material.
Our assumption here is that window shutters are mostly open and facade walls have very different material against walls as pictures are always taken during the day. In the context of hue image, window aperture regions are very different from walls. By estimation of these regions individually, and then verified collectively in horizontal and in vertical, and then further refined by local edges, window apertures are detected.
Balcony Detection
Balconies are also difficult to detect because they do not have prominent features such as shape, color or material. In Haussmannian buildings, wrought iron balconies are mostly seen. And they add another difficulty as they are not plain and solid. Moreover, the bases and consoles of balconies share the same material as the facade wall.
We found here that, wrought iron guards impose a lot of edges in the images. Hence, the activity of a pixel in terms of edge density could be considered as a feature. By using FFT in a sliding window to rank each pixel, we are able to differentiate wrought irons from background walls. Afterward, by using architectural knowledge and searching the edges below wrought iron guards, balcony bases and consoles could also be detected.
Dormer Window Detection
The top section of the facade is normally a mansard roof, which is painted in deep blue or gray, and it could be easily picked up from normal floors. Also dormer windows are widely seen in mansard roofs. To mark the dormer windows, color histogram based segmentation could be used.
Door Entrance Detection
Every building has a prominent door entry for access and security control. Its detection is a problem due to the fact that on the ground floor, various shops exist, which have various colors and shapes. However, the door type and door color are of a limited number, thanks to the building construction regulations. Hence, we could set up a small door database and use color histogram intersection to match and mark them.
Model-Based Reconstruction
We propose a model-based reconstruction of buildings, by utilizing different data with building models which have geometry and semantic definition of buildings. The reason is many folds.
2D Image analysis could help detect the general structure of facades, however, to achieve full facade reconstruction with high fidelity, one needs to use models to detect more details. For example, the decorations and small engraving on the facades could not be easily detected with current image analysis methods. However, buildings are man-made organized works. They have some basic arrangements specific to the architectural knowledge and construction practice.
Also one disadvantage of 2D image analysis is the depth information. This kind of information could only be inferred from range scans in our case. Furthermore, the detailed shape modeling of architectural elements could neither be generated from 2D analysis.
Therefore, in order to achieve full 3D representation of buildings, all sources of data should be fused with the definition of building models in many ways.
2D Facade Model and Matching with Previous Facade Analysis Results
According to Haussmannian Facade characterization study done in the Terra Data project, the 2D spatial arrangement of architectural elements is well defined. In a typical Haussmannian Facade, there are several floors with one mansard roof on the top and one ground floor. On normal floors, there windows are repeated regularly, and on roofs, dormer windows are seen, while on the ground floor, shops are to be found. From the first floor, beyond the ground floor, and until the last floor, reaching the roof, associated balconies are present. On other floors, individual balconies appear. Furthermore, surrounding each window, there are impostes on top, balcony base on the bottom, decorative walls on both shoulders. Those arrangement defines the basic structure of a typical Haussmannian Building.
We have deformed the 2D model to fit previous facade analysis result, so that they are well registered. Moreover, we could extract the decoration on top of the windows. By comparing those image segments to model ones, we are able to determine the decoration type (by optimizing 2D matching with 2D projections of 3D decoration models).
3D Scan Data processing and 3D model adjustment
For the depth information, we obtained it from 3D scans. Manually registered 3D scans were utilized to infer depth by 2D masks generated from 2D Facade analysis. Those information could be fed in an automatic adjustment of manually created 3D building models to achieve a high fidelity modeling.
Conclusion
In conclusion, regarding our goal of Parisian urban environment reconstruction, several achievements have been made, including 2D facade analysis and model-based reconstruction. We are now looking forward to large scale modeling of facades.
