Best tools for multimedia reporting Fri, 14 Nov 2014 18:50:46 +0000 en-US hourly 1 https://s0.wp.com/i/webclip.png 32 32 239681374 https://geojournalism.org/2013/07/cleaning-data-with-refine/ Michael Bauer Wed, 17 Jul 2013 22:36:29 +0000 Data data wrangling environmental data https://dev.cardume.art.br/toolkit/?p=94 <p>Open Refine (previously Google Refine) is a data cleaning software that uses your web browser as an interface. This means it will look like it runs on the internet but all your data remains on your machine and you do not need internet connection to work with it. The main aim of Refine is to [&#8230;]</p> <p>The post <a href="https://geojournalism.org/2013/07/cleaning-data-with-refine/">Cleaning Data with Refine</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> <p>Open Refine (previously Google Refine) is a data cleaning software that uses your web browser as an interface. This means it will look like it runs on the internet but all your data remains on your machine and you do not need internet connection to work with it.</p> <p><span id="more-94"></span></p> <p>The main aim of Refine is to help you exploring and cleaning your data before you use it further. It is built for large datasets – so don’t worry as long as your spreadsheets can hold the information, refine can as well.</p> <h2 class="summary-item" id="creating-a-new-project"><a href="#creating-a-new-project">Creating a new Project</a></h2> <p>To work with your data in Refine you need to start a new project.</p> <p><strong>Walkthrough: Creating a Refine project</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> <li><a href="#"> Step 3 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <li>Start Refine – this will open a browser window pointing to <a href="https://127.0.0.1:3333" target="_blank">https://127.0.0.1:3333</a> if this doesn’t happen open the link with your browser directly</li> <li>Create a new project: On the left tab select the “Create Project” tab:<img alt="Cleaning Data with Refine" decoding="async" height="183" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia01.png" style="margin: 15px 610px 0px 0px;" width="170"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Click on “Choose Files” to choose your downloaded file and click on “next” – you can also use the URL to the <a href="https://geojournalism.org/glossary/79/" target="_blank" title="CSV">CSV</a> directly if your data is hosted on the web.</li> <li>You will get a preview on how refine will interpret your data – if you have selected a well formatted CSV or other file: this should be pretty automatic.</li> <li>Review the preview carefully to make sure the data looks right. Double check character encoding. Much, but not all data uses UTF-8 these days, but make sure you don’t see any funny characters in preview.</li> <li>In our Case all the Column have no column headings – since the first row of the file is actually empty.<img alt="Cleaning Data with Refine" class="size-full wp-image-278 aligncenter" decoding="async" fetchpriority="high" height="255" sizes="(max-width: 531px) 100vw, 531px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia03.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia03.png 531w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia03-300x144.png 300w" style="margin-top: 15px;" width="531"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>If you check further down, you’ll notice that the column headings for the data are in rows 5 and 6.</li> <li>Let’s tell refine where to find column information: On the bottom you can find settings on how refine should look for column information.</li> <li>We’ll tell it to ignore the first 5 lines, take the next 2 lines as column labels and ignore one further row:</li> </ol> <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="size-full wp-image-279 aligncenter" decoding="async" height="99" sizes="(max-width: 485px) 100vw, 485px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia04.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia04.png 485w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia04-300x61.png 300w" style="margin-top: 15px;" width="485"/></p> <ol> <li>Great – now the preview looks much nicer.</li> <li>Name your project in the box on the top right side and click on “Create Project”.</li> <li>The project will open in the project view, this is the basic interface you are going to work with: by default refine shows only 10 rows of data, you can change this on the bar above the data rows. Also you can use the navigation on the right to see the next or previous rows.</li> </ol> </div> </div> </div> </div> .et-tabs-container <p></p> <p>You now have successfully created your first refine project. Remember: although it runs in a web-browser, the Refine server is still on your machine – all the data is there (so no worries if you handle sensitive information)</p> <h2 class="summary-item" id="sorting-and-facetting"><a href="#sorting-and-facetting">Sorting and Facetting</a></h2> <p>Once we created our project, let’s go and explore the data and the Refine interface a bit. Using Refine might be intimidating at first, since it seems so different from spreadsheets, once you get used to it you will notice how easily you can do things with it.</p> <p>One of the commonly used functions in spreadsheets is sorting and filtering data – to figure out minima, maxima or things about certain categories. Refine can do the same thing.</p> <p><strong>Walkthrough: Sorting rows</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> <li><a href="#"> Step 3 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <ol> <li>Refines handles data similar to a spreadsheet: you have rows, columns and cells – a cell is a field defined by a row and a column.</li> <li>To sort your rows based on a specific column click on the small downward triangle next to the column.<img alt="Cleaning Data with Refine" decoding="async" height="300" loading="lazy" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia05-154x300.png" style="margin: 15px 610px 0 0;" width="154"/></li> </ol> </ol> </div> <div class="et_slidecontent"> <ol> <li>Select “Sort…” to open the sorting dialog.</li> <li>You can select what to sort the values as and then what order to sort in. (We’ll sort in text, since for now we only have text columns). <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-281" decoding="async" height="359" loading="lazy" sizes="auto, (max-width: 424px) 100vw, 424px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia06.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia06.png 424w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia06-300x254.png 300w" style="margin-top: 15px;" width="424"/></p> </li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Click “OK” and your rows will be sorted based on the column.</li> <li>To undo the sort, click on the column options again, select “sort” then “remove sort”</li> </ol> <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-282" decoding="async" height="319" loading="lazy" sizes="auto, (max-width: 410px) 100vw, 410px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia07.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia07.png 410w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia07-300x233.png 300w" style="margin-top: 15px;" width="410"/></p> </div> </div> </div> </div> .et-tabs-container <p></p> <p>The other frequently used function in Spreadsheets is filtering – in Refine this is called facetting. Facetting in Refine is really powerful – you will see in most of the rest of the Recipe we’ll use facets.</p> <p><strong>Walkthrough: Facetting rows based on a column</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> <li><a href="#"> Step 3 </a></li> <li><a href="#"> Step 4 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <li>Select the column options for the column you want to facet with</li> <li>Select “Facet”<img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-283" decoding="async" height="300" loading="lazy" sizes="auto, (max-width: 333px) 100vw, 333px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia08.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia08.png 333w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia08-300x270.png 300w" style="margin-top: 15px;" width="333"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>You can facet differently for text, numbers or dates – let’s facet as text – click on “Text facet”</li> <li>This will open a facet in the left bar<img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-284" decoding="async" height="385" loading="lazy" sizes="auto, (max-width: 301px) 100vw, 301px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia09.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia09.png 301w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia09-234x300.png 234w" style="margin-top: 15px;" width="301"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Now select one or more of the choices and you’ll see how your data rows are limited to just those selected.</li> <li>Of course you can add more than one facet and thus filter more than once.</li> <li>Let’s add another facet: Note that one of the columns contain dates: we can add a timeline facet to let us select a date range <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-285" decoding="async" height="300" loading="lazy" sizes="auto, (max-width: 342px) 100vw, 342px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia10.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia10.png 342w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia10-300x263.png 300w" style="margin-top: 15px;" width="342"/></p> </li> </ol> </div> <div class="et_slidecontent"> <ol> <li>This will open a new facet looking like: <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-286" decoding="async" height="173" loading="lazy" sizes="auto, (max-width: 302px) 100vw, 302px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia11.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia11.png 302w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia11-300x171.png 300w" style="margin-top: 15px;" width="302"/></p> </li> </ol> <ol> <li>You can select a time frame by moving the indicators left and right.</li> </ol> </div> <div class="et_slidecontent"> </div> </div> </div> </div> .et-tabs-container <p></p> <h2 class="summary-item" id="reconciling-columns"><a href="#reconciling-columns">Reconciling Columns</a></h2> <p>Sometimes humans make mistakes when they enter data – they mistype city names or put in characters they can not see but the computer can. (For example, you can add a simple space at the end of a name and the computer will think they are different). For this let’s create a text facet for the cities:</p> <p><strong>Walkthrough: Reconciling Columns</strong><br/> </p><div class="et-tabs-container et_sliderfx_fade et_sliderauto_false et_sliderauto_speed_5000 et_slidertype_top_tabs"> <ul class="et-tabs-control"> <li><a href="#"> Step 1 </a></li> <li><a href="#"> Step 2 </a></li> </ul> <!-- .et-tabs-control --> <div class="et-tabs-content"> <div class="et-tabs-content-wrapper"> <div class="et_slidecontent"> <ol> <li>Create a text facet for the City column</li> <li>Scroll down where it says La Paz: see how many different ways there are to write la paz?<img alt="Cleaning Data with Refine" decoding="async" height="181" loading="lazy" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia12.png" style="margin-top: 15px;" width="297"/></li> </ol> </div> <div class="et_slidecontent"> <ol> <li>Let’s fix this! This is so common that refine has a built in mechanism for detecting these kind of differences.</li> <li>Click on the “Cluster” button on the top right in the facet. <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter wp-image-288" decoding="async" height="367" loading="lazy" sizes="auto, (max-width: 591px) 100vw, 591px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia13.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia13.png 739w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia13-300x186.png 300w" style="margin-top: 15px;" width="591"/></p> </li> </ol> <ol> <li>Now check all the things Refine correctly noticed as similar</li> <li>Click on the “Merge Selected &amp; Re-Cluster” button. Play with the keying function and methods on top to identify further possible merges.</li> </ol> </div> </div> </div> </div> .et-tabs-container <p></p> <h2 class="summary-item" id="making-city-names-look-nice"><a href="#making-city-names-look-nice">Making city names look nice</a></h2> <p>Did you notice how most of the Cities are all uppercase? It’s rare to read them like this. And maybe you want to have nicer looking names: No problem. Refine supports this.</p> <p><strong>Walkthrough: Changing Case in Refine</strong></p> <ol> <li>Let’s change the case in our city column from all uppercase to titlecase</li> <li>To do this, open the column options, go to edit cells -&gt; common transforms -&gt; to titlecase <p style="text-align: center;"><img alt="Cleaning Data with Refine" class="aligncenter size-full wp-image-289" decoding="async" height="475" loading="lazy" sizes="auto, (max-width: 516px) 100vw, 516px" src="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia14.png" srcset="https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia14.png 516w, https://geojournalism.org/wp-content/uploads/2013/07/CleaningDatawithRefine-Bolivia14-300x276.png 300w" style="margin-top: 15px;" width="516"/></p> </li> </ol> <ol> <li>Tada – magically your names have been converted.</li> </ol> <p><strong>Congratulations! </strong>You successfully cleaned up a dataset using Refine!</p> <div class="author-shortcodes"> <div class="author-inner"> <div class="author-image"> <img alt="" src="https://farm9.staticflickr.com/8321/7937480904_6f29810da7.jpg"/> <div class="author-overlay"></div> </div> <!-- .author-image --> <div class="author-info"> <strong>Michael</strong> lives in Vienna and works as a Data Wrangler with the Open Knowledge Foundation mostly around the School of Data. After a detour in biomedical research, where he learned to love datawrangling, he spent some time doing advocacy for his passion: freedom in the digital age. He joined the Open Knowledge Foundation to satisfy his curiosity. He will gladly jump on any topic that you point him to. – See more at: <a href="https://okfn.org/about/team/#Michael_Bauer_8212_Data_Wrangler_School_of_Data" target="_blank">https://okfn.org/about/team/#Michael_Bauer_8212_Data_Wrangler_School_of_Data</a> </div> <!-- .author-info --> </div> <!-- .author-inner --> </div> .author-shortcodes <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/> <p>The post <a href="https://geojournalism.org/2013/07/cleaning-data-with-refine/">Cleaning Data with Refine</a> appeared first on <a href="https://geojournalism.org">geojournalism.org</a>.</p> 94