Magento is the largest e-commerce installations world-wide. TransPacific Software being leader in jewelry diamond portals; routinely receive requests to integrate Rapnet or IDEX diamond data feeds with existing Magento based jewelry websites.
However unlike Opencart or Prestashop, integrating Rapnet feeds (often going into 50,000+ diamond records) is complex programming. It took us considerable time and development effects to perfect it.
a) Type of Rapnet Subscription On receiving project proposal to integrate Rapnet data feed with Magento the initial check is on client's Rapnet subscription to determine if they have Rapnet DLS services subscribed or not. For pulling data in CSV format from Rapnet and re inserting in Magento (or in any shopping cart database) Rapnet Diamond Listing Services ( DLS) is required. DLS is not switched on by default with subscription. If not available a request has to be put with Rapnet. Rapport/Rapnet also provides inventory link services to pull data directly from its servers . However to have full data control and customized mark-ups DLS is an absolute must.
b) Server configuration
Second aspect is, what configuration server the client has for its Magento hosting. Magento and Rapnet combination requires powerful server config. A Dedicated server or cloud hosting on Amazon or Rackspace is recommended.
Once both the above issues are discussed integration work proceeds.
Client needs to provide us details on what type of diamond they are interested in. For example some may be interested in diamonds from a particular country only. Or some may want to pull stones which are of particular carat weight e.g. 0.3 to 2 carat. Based of client's requirements Rapnet DLS services are logged into and appropriate URL is generated which is supposed to download the diamond data in csv.
Diamond data pulled has lot of impurity as it's uploaded directly by diamond wholesalers from across the globe. TransPacific has perfected filters which weed out lot of inaccurate data. These may be wrong GIA cert numbers or records without weight , shape or empty records etc. After data scrubbing it is ready for reinserting in clients Magento database.
4) Deleting unwanted data
Daily Rapnet data is updated by its members. Appropriately data on Magento needs to be refreshed by removing unavailable or sold out stones. It is a complicated process but perfected by TransPacific Software through its extensively tested scripts.
5) Updating / inserting data
The process of inserting data consist of parsing the csv files and inserting the data under appropriate categories and attributes
6) First time Preparation of database Creating category and subcategories
Initially" Diamond" as main category is created manually.
Inside diamond category sub-categories of shape is created manually for example Round, Pear, Emerald, Heart, Cushion, Asscher, Princess, Marquise, Oval, Radiant
At the time of data insertion we fetch above category ids by passing shape.
Attribute insertion through script:
If the requirement is of large number of attributes and creating them manually is cumbersome we use, Magento function to bulk create attributes: $installer->addAttribute('catalog_product', 'manufacturer', $attribute);
7) Price markup
This is an important functionality which we inbuilt with data insertion. Client provides us his markup percentage (could be flat say 20% on all diamonds or may be staggering as per carat weight)
8) Cron job setting
To eliminate any manual intervention cron job is created to repeat the process of data pulling and inserting at required frequency (daily/ twice in a week / once in a week). A log file is created to maintain the log on cron job executed.
TransPacific has also specialized in AJAX based search interface for diamonds. The search interface filters through thousands of diamond records at blazing fast speed to pull out the filter records. Search interface is extremely user friendly, multi browser compatible and searches Diamonds on all core parameters including carat weight, price, clarity, florescence and shape.
Commonly Occurring issue while integrating rapnet data with Magento Magento Index management.
Issue : Magento indexes most of its data in order to access it faster. When you make changes to your Store such as updating some of your products, you need to re-index the data so that the changes can show up on frontend. Larger database requires more time to re-index the data. So re-indexing data by Magento admin panel could time out with thousands of Diamond records
Solution : There is another way to reindex your magento data and it is to do it by cron jobs. You can set cron job by Cpanel or SSH.Maximum execution time of 30 seconds exceeded.
Issue : Magento's default max_execution_time is of 30 seconds but to insert large data like Rapnet CSV it will often time out.
Solution : There is a way to increase max execution time. Either by in php .ini or in htaccess files. Change max_execution_time=18000.
CSV files read or write permission.
To read or write from CSV file you have to mention the read/write permission to CSV file.
Commonly Occurring issue while integrating rapnet data with magento
Magento Index management.
Issue : Magento indexes most of its data in order to access it faster. When you make changes to your Store such as updating some of your products, you need to re-index the data so that the changes can show up on frontend. Larger database requires more time to re-index the data. So re-indexing data by Magento admin panel could time out with thousands of Diamond records
Solution : There is another way to reindex your magento data and it is to do it by cron jobs. You can set cron job by Cpanel or SSH.
Maximum execution time of 30 seconds exceeded.
Issue : Magento's default max_execution_time is of 30 seconds but to insert large data like Rapnet CSV it will often time out.
Solution : There is a way to increase max execution time. Either by in php .ini or in htaccess files. Change max_execution_time=18000.
CSV files read or write permission.
To read or write from CSV file you have to mention the read/write permission to CSV file.
Transpacific has specialization in integrating Rapnet API diamond feed with Magento. Below is a white paper on how we do it
Update March 2018: Rapnet has discontinued DLS services and replaced it with Rapnet Instant Inventory web services
A Quick technical walkthrough on how we integrated Rapnet Diamond feeds with Magento
rapnet_magento_1.png
Magento is the largest e-commerce installations world-wide. TransPacific Software being leader in jewelry diamond portals; routinely receive requests to integrate Rapnet or IDEX diamond data feeds with existing Magento based jewelry websites.
However unlike Opencart or Prestashop, integrating Rapnet feeds (often going into 50,000+ diamond records) is complex programming. It took us considerable time and development effects to perfect it.
How we do it
rapnet_infograph_2.jpg
1) Initial Analysis
a) Type of Rapnet Subscription
On receiving project proposal to integrate Rapnet data feed with Magento the initial check is on client's Rapnet subscription to determine if they have Rapnet DLS services subscribed or not. For pulling data in CSV format from Rapnet and re inserting in Magento (or in any shopping cart database) Rapnet Diamond Listing Services ( DLS) is required. DLS is not switched on by default with subscription. If not available a request has to be put with Rapnet. Rapport/Rapnet also provides inventory link services to pull data directly from its servers . However to have full data control and customized mark-ups DLS is an absolute must.
b) Server configuration
Second aspect is, what configuration server the client has for its Magento hosting. Magento and Rapnet combination requires powerful server config. A Dedicated server or cloud hosting on Amazon or Rackspace is recommended.
Once both the above issues are discussed integration work proceeds.
rapnet_DLS_3.jpg
2) Data pulling from Rapnet servers in CSV format
Client needs to provide us details on what type of diamond they are interested in. For example some may be interested in diamonds from a particular country only. Or some may want to pull stones which are of particular carat weight e.g. 0.3 to 2 carat.
Based of client's requirements Rapnet DLS services are logged into and appropriate URL is generated which is supposed to download the diamond data in csv.
A typical URL may look like
rapnet_url_4.jpg
3) Filtering data
Diamond data pulled has lot of impurity as it's uploaded directly by diamond wholesalers from across the globe. TransPacific has perfected filters which weed out lot of inaccurate data. These may be wrong GIA cert numbers or records without weight , shape or empty records etc. After data scrubbing it is ready for reinserting in clients Magento database.
4) Deleting unwanted data
Daily Rapnet data is updated by its members. Appropriately data on Magento needs to be refreshed by removing unavailable or sold out stones. It is a complicated process but perfected by TransPacific Software through its extensively tested scripts.
5) Updating / inserting data
The process of inserting data consist of parsing the csv files and inserting the data under appropriate categories and attributes
6) First time Preparation of database Creating category and subcategories
product_with_dia_attri_3.jpg
Initially" Diamond" as main category is created manually.
Inside diamond category sub-categories of shape is created manually for example Round, Pear, Emerald, Heart, Cushion, Asscher, Princess, Marquise, Oval, Radiant
At the time of data insertion we fetch above category ids by passing shape.
Attribute insertion through script:
If the requirement is of large number of attributes and creating them manually is cumbersome we use, Magento function to bulk create attributes: $installer->addAttribute('catalog_product', 'manufacturer', $attribute);
7) Price markup
This is an important functionality which we inbuilt with data insertion. Client provides us his markup percentage (could be flat say 20% on all diamonds or may be staggering as per carat weight)
8) Cron job setting
To eliminate any manual intervention cron job is created to repeat the process of data pulling and inserting at required frequency (daily/ twice in a week / once in a week). A log file is created to maintain the log on cron job executed.
9) On front Creating Ajax based search interface
search_interface_5.jpg
TransPacific has also specialized in AJAX based search interface for diamonds. The search interface filters through thousands of diamond records at blazing fast speed to pull out the filter records. Search interface is extremely user friendly, multi browser compatible and searches Diamonds on all core parameters including carat weight, price, clarity, florescence and shape.
Commonly Occurring issue while integrating rapnet data with Magento Magento Index management.
Issue : Magento indexes most of its data in order to access it faster. When you make changes to your Store such as updating some of your products, you need to re-index the data so that the changes can show up on frontend. Larger database requires more time to re-index the data. So re-indexing data by Magento admin panel could time out with thousands of Diamond records
Solution : There is another way to reindex your magento data and it is to do it by cron jobs. You can set cron job by Cpanel or SSH.Maximum execution time of 30 seconds exceeded.
Issue : Magento's default max_execution_time is of 30 seconds but to insert large data like Rapnet CSV it will often time out.
Solution : There is a way to increase max execution time. Either by in php .ini or in htaccess files. Change max_execution_time=18000.
CSV files read or write permission.
To read or write from CSV file you have to mention the read/write permission to CSV file.
10) Here is a live demo of Rapnet diamond Integrated on Magento
rapnet_idex.jpg
Commonly Occurring issue while integrating rapnet data with magento
Magento Index management.
Issue : Magento indexes most of its data in order to access it faster. When you make changes to your Store such as updating some of your products, you need to re-index the data so that the changes can show up on frontend. Larger database requires more time to re-index the data. So re-indexing data by Magento admin panel could time out with thousands of Diamond records
Solution : There is another way to reindex your magento data and it is to do it by cron jobs. You can set cron job by Cpanel or SSH.
Maximum execution time of 30 seconds exceeded.
Issue : Magento's default max_execution_time is of 30 seconds but to insert large data like Rapnet CSV it will often time out.
Solution : There is a way to increase max execution time. Either by in php .ini or in htaccess files. Change max_execution_time=18000.
CSV files read or write permission.
To read or write from CSV file you have to mention the read/write permission to CSV file.