Advanced Filtering for File Metadata is a Premium License feature, currently in early access, introduced in File IQ version 10.2. This feature allows users, including storage and IT administrators, to filter the file metadata discovered by File IQ.
With Advanced Filtering for File Metadata, users can easily identify a specific set of files that meet particular criteria.
This feature is beneficial for tasks such as:
Finding the 10 largest files.
Locating files owned by a specific user.
Identifying files that contain "projectA" in the file path.
Important: Due to early access, certain functionality might be subject to change. You might encounter limitations or unexpected behavior.
Advanced Filtering Feature Configuration
Advanced Filtering involves a two-step setup process. First, you need to enable the feature. Next, you must wait for the volumes to be scanned and then indexed. After that, Advanced Filtering is available for the specific volumes you want. Each volume can be enabled or disabled individually.
To configure the File IQ Advanced Filtering, follow these steps:
Navigate to the File IQ UI.
Click Configuration > Advanced Filtering.
Check the Enable File IQ Metadata Filtering box.
Click Save.
Configuring File Metadata Filtering for a Volume
The second component of the configuration involves enabling your volumes for Advanced Filtering. This section outlines that process.
Note: If you do not see a volume you have recently added, wait until its scan is complete. You can use the Service Support dashboard to monitor Volume scans.
To configure a volume for Advanced Filtering, follow these steps:
Navigate to the File IQ UI.
Click Configuration > Advanced Filtering.
Under Volume Settings, check the box for the volume, or volumes, for which you want Advanced Filtering.
Click Edit (X) Volumes.
Click the Enabled toggle to On.
Click Save.
Volume Indexing
Advanced filtering is available only for indexed volumes. Before you can use advanced filtering, a volume must be indexed first.
Volumes are indexed after scanning is complete, which usually occurs once a day. Volume indexes are updated daily.
A maximum of two volumes can be indexed simultaneously. If additional indexing tasks are pending, they are queued and processed as soon as the previous index tasks are completed.
Note: The current volume indexing process does not utilize File IQ’s incremental scan capabilities. As a result, each indexing run takes approximately the same amount of time as the initial indexing, even if there have been little changes to the underlying volume.
Monitoring Volume Indexing
To monitor volume indexing, open the Service Support dashboard in Grafana and scroll down to the Volume Indexing panel. This panel lists all volumes, whether they are enabled for indexing, and their current indexing state.
The Volume Indexing panel provides the following volume information:
Column | Description |
|---|---|
Volume Name | UniFS Volume name.. |
Indexing Enabled | Yes, if this volume is enabled for indexing, No otherwise |
Status | Current indexing status. See status definitions below. |
Scheduled | Time when the volume was scheduled for volume indexing (in the timezone specified in the time range selection drop-down in the top right corner of the dashboard). |
Started | Time when the volume indexing process started. |
Finished | Time when the volume indexing process completed. |
Execution Status | Total volume indexing execution time. |
Message | Error message, if one is present. |
The indexing status can be one of the following:
Status | Description |
|---|---|
Pending | This indicates that the volume is about to be indexed. |
Indexing | This volume is currently being indexed. |
Succeeded | The volume indexing execution completed successfully. |
Failed | An error occurred when indexing this volume. Refer to the Message column for more information. |
Not_Configured | This volume is not enabled for Advanced Filtering. You can enable this volume for Advanced Filtering in the File IQ user interface, located by clicking Configuration > Advanced Filtering. |
Skipped | If a new scan finishes before the previous scan has been indexed, a new indexing task is created for the latest scan. In this case, the system skips indexing the older scan and proceeds directly with indexing the most recent one. |
Volume Indexing History
Below the Volume Indexing panel is the Volume Indexing History panel. This panel includes the same information as the Volume Indexing panel, along with a timestamp column and an additional status option. By default, this panel is collapsed.
Volume indexing history is retained for one year and automatically deleted after this period.
File Metadata Advanced Filtering Dashboard
Volume indexing must occur before the volume appears in the Volume Name drop-down menu and is available for use in Advanced filtering. Volumes are scheduled for indexing after the scanning process is complete, which usually occurs once a day. Therefore, volume indexes are updated daily.
Dashboard Layout
The following numbers correspond to the dashboard components:
Number | Component | Description |
|---|---|---|
1 | Volume Name | Select the volume to filter. |
2 | File/Directory Limit | Select the number of rows to return in the result table. This value only impacts the result table and not the three statistics sections. |
3 | Data Generated At | This displays the date the volume was last indexed. |
4 | Advanced Filtering | Enter your filter criteria |
5 | Reset All / Apply | Use these buttons to reset all filter criteria or proceed with the filter. |
6 | Total File Count | Statistic section displaying the total number of matching files, irrespective of the specified File / Directory Limit. |
7 | Total File Size | Statistic section displaying the total size of matching files, irrespective of the specified File / Directory Limit. |
8 | Total Directory Count | Statistic section displaying the total number of matching directories, irrespective of the specified File / Directory Limit. |
9 | Message | Results table. This table displays up to the File / Directory Limit rows. This table also displays error messages for invalid selections. |
10 | Indexing Status | Open the Service Support Dashboard, where you can view the Volume Indexing status in the Volume Indexing panel. |
11 | Help | Link to the Advanced Filtering documentation. |
Dashboard Results
After running a query, the results display in the table below Messages. Clicking a file or directory opens the Volume Explorer dashboard on the selected volume and file or directory.
The Results table provides the following information:
Column | Description |
|---|---|
Rank | Rank of the entry in the result set in the sort order specified in the Sort Order section of the Advanced Filtering criteria panel. |
Type | An icon indicating whether the item is a file or a directory. |
File Name | Full path and file name. |
Size | The size of the file or directory. For a directory, this includes the total size of all its contents, including subdirectories. |
Created | The date this file or directory was created. |
Modified | The date this file or directory was last updated. |
Accessed | The date this item was last accessed. * |
Files | For directories: The total number of files in the directory, including all files in its subdirectories. |
Dirs | For directories: The total number of subdirectories, including all nested subdirectories. |
Owner | The owner of this file or directory. |
Group | The group owner of this file or directory. |
*Important: Access times for files and directories are updated when audit events are received from a Network Edge Appliance (NEA) running version 10.2 or later. To ensure accurate updates, configure all NEAs connected to the filtered volume to send audit events to this File IQ appliance. Note that access times for files and directories accessed before upgrading the NEAs to version 10.2 are unavailable. In such cases where access time is not available, the modified time is used for filtering and display purposes.
Using the Advanced Filtering Dashboard
To use the Advanced Filtering dashboard, follow these steps:
From the Dashboards Homepage, select the Advanced Filtering - File Metadata dashboard.
Navigate to the Volume Name field, and choose a volume from the drop-down.
From the File/Directory Limit drop-down, select the number of items to display in the result table.
Note: You can display up to 10,000 items.
Note: The Total File Count, Total File Size, and Total Directory Count display statistics for all files and directories that match your search criteria, irrespective of the File/Directory Limit.
Select your filter criteria from the filters located on the left-hand side. Expand and collapse sections as required.
Use the following tips to get the most out of your filtering:
Any field included as part of the filter has a blue tick next to it and next to its section header.
Basic wildcards (* and ?) are accepted for all text strings.
Empty or zero-number fields are not included in the filter.
The sort order you select determines which top items appear in the filter results, making it helpful in retrieving specific "Top N" files and directories, such as the top 10 largest files. The items displayed in the results table are ranked based on the chosen sort order. Once the results are displayed, they can be sorted independently according to the table's settings, which can be adjusted by clicking a column header. The default sort order is based on the Rank column. If you click a different column to sort the table, the Rank column is no longer used for sorting, and the list might not appear in the original ranked order defined by your selected sort option.
Click Reset All at any time to reset all filters and sort order to their default values.
Click Apply.
(Optional) To download your results, follow these steps:
Click the three-dot button on the top right of the table.
Select Inspect > Data.
Select whether you want formatted or raw data, and if you want to download an Excel-compatible CSV file.
Click Download CSV.
Example Filter Scenarios
The table below shows example filter configurations for common search scenarios. Use these examples as a starting point to customize your search.
Scenario | Configuration |
|---|---|
Find all PDF files | File Name: *.pdf |
Find all PDF files over 1MB | File Name: *.pdf File Size: Minimum: 1 MB |
Find the 10 biggest bin files | File/Directory Limit: 10 File Name: *.bin Sort Order: File size (Large to Small) |
Find all files with projectA in the path | File Path: projectA |
Find docx files not modified in the last 90 days | File Name: *.docx Modified: Before 90 days |
Find all files owned by userA | Owner: userA |
Find all files owned by userA that have not been modified in 6 months | Owner: userA Modified: Before 6 months |
Find the 2 biggest files created over 6 months ago and not accessed in the last 3 months | File/Directory Limit: 2 Created: Before 6 months Accessed: Before 3 months *See filter restrictions below. Sort Order: File size (Large to Small) |
*Important: Access times for files and directories are updated when audit events are received from a Network Edge Appliance (NEA) running version 10.2 or later. To ensure accurate updates, configure all NEAs connected to the filtered volume to send audit events to this File IQ appliance. Note that access times for files and directories accessed before upgrading the NEAs to version 10.2 are unavailable. In such cases where access time is not available, the modified time is used for filtering and display purposes.
Unsupported Indexing and Filtering Scenarios
This dashboard filters only file and directory metadata, not audit events. For example, you cannot filter for all files accessed or modified by a specific user, as this would require filtering all past audit events.
File IQ currently does not retrieve complete access control lists (ACLs) for individual files and folders. As a result, filtering based on user or group access permissions is not supported at this time.
Troubleshooting
Advanced Filtering is not Available in the File IQ UI
If you do not see the Advanced Filtering menu option in the File IQ user interface and you are using File IQ version 10.2, it indicates that this feature has not been enabled for your account. If you are a File IQ Premium user, reach out to your Nasuni account representative to discuss the availability of this feature.
Advanced Filtering Configuration Page Shows “Nasuni File IQ Premium not Licensed”
Advanced Filtering is a premium feature and is not included in the File IQ Basic plan.
To begin using this feature, contact your Nasuni account representative. They can guide you through the licensing process, answer any questions you may have, and ensure a smooth transition to accessing this premium feature.
Memory Usage
File IQ might require additional memory for Advanced Filtering.
We recommend checking the memory requirements in the File IQ UI under Configuration > Retention Configuration to confirm them.
Lengthy Indexing Times
The following factors usually drive indexing time:
Volume size: Large volumes (over 100 million files) might take several hours to index.
VM memory size: Ensure it is suitably sized by navigating to Configuration > Retention Configuration in the File IQ UI.
File IQ DB disk specifications: Since both the File IQ database and index are on the same File IQ DB disk, you might need to increase IOPS or throughput to ensure adequate performance. You can use your cloud provider’s virtual machine and disk monitoring performance charts to monitor disk utilization. Also, ensure that the File IQ DB disk meets the performance requirements specified in the sizing spreadsheet.