BMEcat import / export

Hey hey,

I have to import and export a special XML catalog format called BMEcat.

My actual solution is to use w-vision DataDefinitions Bundle and extend it with my own BMEcatProvider based on the XmlProvider

But as BMEcat is kinda common in german regions I see a small chance that there might be someone who has done it before and could share his experience and or his code.

I did a very basic BMECat with DataDefinitions once, but that was years ago and really basic.

You could create a BMECat Provider for DataDefinitions and standardize it so we could merge it into the Bundle directly. WDYT?

1 Like

I think that would be a great idea.

I will get in contact with you in the next few days via pn.

Hi Lukas, a BMECat connector is a great idea, but unfortunately a BMECat (1.2, 2005) with or without ETIM, eclass etc. looks always little bit different.

But it is not too difficult to parse a BMECat using a CLI skript

Sample code for an existing 1.2 importer

$workingDirectory = getcwd();
chdir(__DIR__);
include ("../../../../vendor/autoload.php");
\Pimcore\Bootstrap::startupCli();
chdir($workingDirectory);

//this is optional, memory limit could be increased further (pimcore default is 1024M)
ini_set('memory_limit', '2048M');
ini_set("max_execution_time", "33600");

//execute in admin mode
define("PIMCORE_ADMIN", true);

$disableCache = true;

$filename = 'BMEcat_Preisliste_2019_02_V06_19062019.xml';

$xml = simplexml_load_file($filename, 'SimpleXMLElement', LIBXML_NOCDATA);

...


	$Products = $xml->xpath('/BMECAT/T_NEW_CATALOG')[0];

	  echo "*** Import Products ***\n";

		foreach ($Products->ARTICLE as $Product ) {

			$SUPPLIER_AID = (string) $Product->xpath('SUPPLIER_AID')[0];
			$DESCRIPTION_SHORT = (string) $Product->xpath('ARTICLE_DETAILS/DESCRIPTION_SHORT')[0];
			$DESCRIPTION_LONG = (string) $Product->xpath('ARTICLE_DETAILS/DESCRIPTION_LONG')[0];
			$EAN = (string) $Product->xpath('ARTICLE_DETAILS/EAN')[0];
			$MANUFACTURER_AID = (string) $Product->xpath('ARTICLE_DETAILS/MANUFACTURER_AID')[0];
			$MANUFACTURER_TYPE_DESCR = (string) $Product->xpath('ARTICLE_DETAILS/MANUFACTURER_TYPE_DESCR')[0];
			$ERP_GROUP_SUPPLIER = (string) $Product->xpath('ARTICLE_DETAILS/ERP_GROUP_SUPPLIER')[0];
			$REMARKS = (string) $Product->xpath('ARTICLE_DETAILS/REMARKS')[0];
			$FEATURES = $Product->xpath('ARTICLE_FEATURES');
			$REFERENCES = $Product->xpath('ARTICLE_REFERENCE');
...

			$part = \Pimcore\Model\DataObject\Produkt::getByProduktnummer($SUPPLIER_AID,1);

			if (is_null($part)) {
				echo "... Create part in pimcore " . $SUPPLIER_AID . "\n";
				$part = new Pimcore\Model\DataObject\Produkt();
				$part->setCreationDate(time());
				$part->setPublished(true);
				$part->setKey(makeKey($SUPPLIER_AID));
				$part->setParentId(2);
			} else {
				echo "... Update part in pimcore " . $SUPPLIER_AID . "\n";
			}

			$part->setRefcode($MANUFACTURER_TYPE_DESCR);
			$part->setProduktnummer($SUPPLIER_AID);
			$part->setEAN($EAN);
			$part->setName($DESCRIPTION_SHORT, "de");

			$PriceList = $Product->xpath('ARTICLE_PRICE_DETAILS')[0];
			if ($PriceList->ARTICLE_PRICE) {
				echo "...... Update prices\n";
	      $PricesFieldCollection  = new DataObject\Fieldcollection();
			  foreach ($PriceList->ARTICLE_PRICE as $Price ) {
					$PriceAmount = (string) $Price->xpath('PRICE_AMOUNT')[0];
					$PriceCurrencyCode = (string) $Price->xpath('PRICE_CURRENCY')[0];
					$PriceCountry = 'DE';
					$PriceType = (string) xml_attribute($Price, 'price_type');
					$PriceItem = new DataObject\Fieldcollection\Data\Preis();
					$PriceItem->setLand($PriceCountry);
					$PriceItem->setTyp($PriceType);
					$PriceItem->setWaehrung($PriceCurrencyCode);
					$PriceItem->setPreis($PriceAmount);
				...	$PricesFieldCollection->add($PriceItem);
					$PriceItem = null;
				}
	      if ($PricesFieldCollection) $part->setPreise($PricesFieldCollection);
	      $PricesFieldCollection = null;
			}

Maybe this is helpful for your project?

Cheers
Markus

1 Like

Hey @incoxx,

thank you very much for sharing! I think this piece of code will be really helpful for me !

As you mentioned parsing the BMEcat XML File shouldn’t be the problem but the fact that not every BMEcat looks like the same is pretty unhandy and will need a lot of filtering which hopefully won’t make the process to slow.

I think most importantly is to somehow select which element (eg: CATALOG, SUPPLIER, etc) you want to import, from there you only have to flatten the data to make it selectable in DataDefinitions. I would make something like:

SOURCE BME (from wikipedia):

<BMECAT version="2005" xmlns="http://www.bmecat.org/bmecat/2005fd">
    <HEADER>
        <CATALOG>
            <LANGUAGE>eng</LANGUAGE>
            <CATALOG_ID>QA_CAT_002</CATALOG_ID>
            <CATALOG_VERSION>001.002</CATALOG_VERSION>
            <CATALOG_NAME>Office Material</CATALOG_NAME>
            <DATETIME type="generation_date">
                <DATE>2004-08-20</DATE>
                <TIME>10:59:54</TIME>
                <TIMEZONE>-02:00</TIMEZONE>
            </DATETIME>
            <CURRENCY>EUR</CURRENCY>
        </CATALOG>
        <BUYER>
            <BUYER_ID type="buyer_specific">aggibuyer</BUYER_ID>
            <BUYER_NAME>BuyAll Corp.</BUYER_NAME>
            <ADDRESS type="buyer">
                <NAME>BuyAll Corp.</NAME>
                <CONTACT>Bill Smith</CONTACT>
            </ADDRESS>
        </BUYER>
        <SUPPLIER>
            <SUPPLIER_NAME>Office Supplies AG</SUPPLIER_NAME>
        </SUPPLIER>
    </HEADER>
    <T_NEW_CATALOG>
        <PRODUCT mode="new">
            <SUPPLIER_PID type="supplier_specific">Q20-P09</SUPPLIER_PID>
            <PRODUCT_DETAILS>
                <DESCRIPTION_SHORT>Post-Safe Polythene Envelopes</DESCRIPTION_SHORT>
                <DESCRIPTION_LONG>All-weather lightweight envelopes protect your contents and save you money.
                                  ALL-WEATHER. Once sealed, Post-Safe envelopes are completely waterproof. Your
                                  contents won't get damaged.</DESCRIPTION_LONG>
                <INTERNATIONAL_PID type="ean">9783161484100</INTERNATIONAL_PID>
                <MANUFACTURER_NAME>Concurrent Limited</MANUFACTURER_NAME>
            </PRODUCT_DETAILS>
            <PRODUCT_FEATURES>
                <REFERENCE_FEATURE_SYSTEM_NAME>UNSPSC-5.02</REFERENCE_FEATURE_SYSTEM_NAME>
                <REFERENCE_FEATURE_GROUP_ID>44121505</REFERENCE_FEATURE_GROUP_ID>
            </PRODUCT_FEATURES>
            <PRODUCT_ORDER_DETAILS>
                <ORDER_UNIT>1</ORDER_UNIT>
                <CONTENT_UNIT>Stk</CONTENT_UNIT>
            </PRODUCT_ORDER_DETAILS>
            <PRODUCT_PRICE_DETAILS>
                <PRODUCT_PRICE price_type="net_list">
                    <PRICE_AMOUNT>16.49</PRICE_AMOUNT>
                    <PRICE_CURRENCY>EUR</PRICE_CURRENCY>
                    <LOWER_BOUND>1</LOWER_BOUND>
                </PRODUCT_PRICE>
                <PRODUCT_PRICE price_type="net_list">
                    <PRICE_AMOUNT>11.49</PRICE_AMOUNT>
                    <PRICE_CURRENCY>EUR</PRICE_CURRENCY>
                    <LOWER_BOUND>50</LOWER_BOUND>
                </PRODUCT_PRICE>
            </PRODUCT_PRICE_DETAILS>
            <PRODUCT_PRICE_DETAILS>
                <PRODUCT_PRICE price_type="net_customer">
                    <PRICE_AMOUNT>10.29</PRICE_AMOUNT>
                    <PRICE_CURRENCY>EUR</PRICE_CURRENCY>
                    <LOWER_BOUND>1</LOWER_BOUND>
                </PRODUCT_PRICE>
                <PRODUCT_PRICE price_type="net_customer">
                    <PRICE_AMOUNT>9.29</PRICE_AMOUNT>
                    <PRICE_CURRENCY>EUR</PRICE_CURRENCY>
                    <LOWER_BOUND>50</LOWER_BOUND>
                </PRODUCT_PRICE>
            </PRODUCT_PRICE_DETAILS>
            <MIME_INFO>
                <MIME>
                    <MIME_TYPE>image/gif</MIME_TYPE>
                    <MIME_SOURCE>P09.gif</MIME_SOURCE>
                    <MIME_PURPOSE>normal</MIME_PURPOSE>
                    <MIME_ORDER>1</MIME_ORDER>
                </MIME>
                <MIME>
                    <MIME_TYPE>url</MIME_TYPE>
                    <MIME_SOURCE>http://www.bmecat.org</MIME_SOURCE>
                    <MIME_PURPOSE>others</MIME_PURPOSE>
                    <MIME_ORDER>2</MIME_ORDER>
                </MIME>
            </MIME_INFO>
        </PRODUCT>
    </T_NEW_CATALOG>
</BMECAT>

to

ELEMENT: T_NEW_CATALOG

PRODUCT: ROW 1

supplier_pid: Q20-P09
supplier_pid_type: supplier_specific
product_details_description_short: ...
product_details_description_long: ...
product_features_reference_feature_system_name: UNSPSC-5.02
product_price_details: array
mime_info: array

product_price_details and mime_info as array, cause the source data is already an array. Everything else can be associative.

Not sure if that is easily possible… But that’d be my approach to a sort of generic solution

1 Like

Yep, that’s definitely true.
The other problem i see is to somehow generate a header declaration to make mapping possible. This has to be done in a pre-run I think, because there is nothing like a first row in csv’s which can be copied by the user in seconds.

Example File would be a way to go. Like DataDefs does with Excel

I had some experience importing BMECat using Data Definitions. Beside provider that flattens the data, we use filter to import ETIM classification into Classification store fields.

Some remarks:

  • keep in mind differences between versions of bmecat (ARTICLE vs PRODUCT as a main node to loop)
  • importing media from mime fields needed creating custom interpreters
  • not every product node has the same attributes set (looping through multiple product nodes is needed to collect headers data)
  • different suppliers can deliver differently formatted files, in practice it’s needed to create separate mapping for each supplier.
  • use iterator to speed up the import process and lower the memory consumption

If we would like to create the OS implementation of BMECat provider, we should also include ETIM Classification importer, that would implement ETIM in a compatible format. Maybe this should be a seperate bundle, that integrates with Data Definitions and Process Manager?

2 Likes

@jplaskonka That sounds like a good idea as well. Maybe we can create a real BMECat Bundle and properly integrate it into DataDefs.

Is there any schema for a BMECat file?

1 Like

The specifications are available for free on the official website: