Package org.apache.drill.exec.store.parquet

Class BaseParquetMetadataProvider

java.lang.Object

org.apache.drill.exec.store.parquet.BaseParquetMetadataProvider

All Implemented Interfaces:

ParquetMetadataProvider, TableMetadataProvider

Direct Known Subclasses:

ParquetTableMetadataProviderImpl

public abstract class BaseParquetMetadataProvider
extends Object
implements ParquetMetadataProvider

Implementation of ParquetMetadataProvider which contains base methods for obtaining metadata from parquet statistics.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	BaseParquetMetadataProvider.Builder<T extends BaseParquetMetadataProvider.Builder<T>>

Field Summary

Fields

Modifier and Type	Field	Description
protected final List<ReadEntryWithPath>	entries
protected Set<org.apache.hadoop.fs.Path>	fileSet
static final Object	NULL_VALUE	HashBasedTable cannot contain nulls, used this object to represent null values.
protected MetadataBase.ParquetTableMetadataBase	parquetTableMetadata
protected final ParquetReaderConfig	readerConfig
protected TupleMetadata	schema
protected DrillStatsTable	statsTable
protected final org.apache.hadoop.fs.Path	tableLocation
protected final String	tableName

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BaseParquetMetadataProvider(BaseParquetMetadataProvider.Builder<?> builder)

Method Summary

Modifier and Type	Method	Description
boolean	checkMetadataVersion()
List<ReadEntryWithPath>	getEntries()	Returns list of ReadEntryWithPath instances which represents paths to files to be scanned.
FileMetadata	getFileMetadata(org.apache.hadoop.fs.Path location)
Set<org.apache.hadoop.fs.Path>	getFileSet()	Returns list of file locations for table.
List<FileMetadata>	getFilesForPartition(PartitionMetadata partition)
Map<org.apache.hadoop.fs.Path,FileMetadata>	getFilesMetadataMap()
List<org.apache.hadoop.fs.Path>	getLocations()	Returns list of file paths which belong to current table.
NonInterestingColumnsMetadata	getNonInterestingColumnsMetadata()
List<SchemaPath>	getPartitionColumns()
List<PartitionMetadata>	getPartitionMetadata(SchemaPath columnName)
List<PartitionMetadata>	getPartitionsMetadata()
List<RowGroupMetadata>	getRowGroupsMeta()	Returns list of RowGroupMetadata instances which provides metadata for specific row group and its columns.
com.google.common.collect.Multimap<org.apache.hadoop.fs.Path,RowGroupMetadata>	getRowGroupsMetadataMap()	Returns multimap of RowGroupMetadata instances which provides metadata for specific row group and its columns mapped to their locations.
Map<org.apache.hadoop.fs.Path,SegmentMetadata>	getSegmentsMetadataMap()
TableMetadata	getTableMetadata()
protected void	init(BaseParquetMetadataProvider metadataProvider)
void	initializeMetadata()	Method which initializes all metadata kinds to get rid of parquetTableMetadata.
protected abstract void	initInternal()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

NULL_VALUE

public static final Object NULL_VALUE

HashBasedTable cannot contain nulls, used this object to represent null values.

entries

protected final List<ReadEntryWithPath> entries

readerConfig

protected final ParquetReaderConfig readerConfig

tableName

protected final String tableName

tableLocation

protected final org.apache.hadoop.fs.Path tableLocation

parquetTableMetadata

protected MetadataBase.ParquetTableMetadataBase parquetTableMetadata

fileSet

protected Set<org.apache.hadoop.fs.Path> fileSet

schema

protected TupleMetadata schema

statsTable

protected DrillStatsTable statsTable

Constructor Details

BaseParquetMetadataProvider

protected BaseParquetMetadataProvider(BaseParquetMetadataProvider.Builder<?> builder)

Method Details

init

protected void init(BaseParquetMetadataProvider metadataProvider)
             throws IOException

Throws:

IOException

initializeMetadata

public void initializeMetadata()

Method which initializes all metadata kinds to get rid of parquetTableMetadata. Once deserialization and serialization from/into metastore classes is done, this method should be removed to allow lazy initialization.

getNonInterestingColumnsMetadata

public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata()

Specified by:

getNonInterestingColumnsMetadata in interface TableMetadataProvider

getTableMetadata

public TableMetadata getTableMetadata()

Specified by:

getTableMetadata in interface TableMetadataProvider

getPartitionColumns

public List<SchemaPath> getPartitionColumns()

Specified by:

getPartitionColumns in interface TableMetadataProvider

getPartitionsMetadata

public List<PartitionMetadata> getPartitionsMetadata()

Specified by:

getPartitionsMetadata in interface TableMetadataProvider

getPartitionMetadata

public List<PartitionMetadata> getPartitionMetadata(SchemaPath columnName)

Specified by:

getPartitionMetadata in interface TableMetadataProvider

getFileMetadata

public FileMetadata getFileMetadata(org.apache.hadoop.fs.Path location)

Specified by:

getFileMetadata in interface TableMetadataProvider

getFilesForPartition

public List<FileMetadata> getFilesForPartition(PartitionMetadata partition)

Specified by:

getFilesForPartition in interface TableMetadataProvider

getSegmentsMetadataMap

public Map<org.apache.hadoop.fs.Path,SegmentMetadata> getSegmentsMetadataMap()

Specified by:

getSegmentsMetadataMap in interface TableMetadataProvider

getFilesMetadataMap

public Map<org.apache.hadoop.fs.Path,FileMetadata> getFilesMetadataMap()

Specified by:

getFilesMetadataMap in interface TableMetadataProvider

getEntries

public List<ReadEntryWithPath> getEntries()

Description copied from interface: ParquetMetadataProvider

Returns list of ReadEntryWithPath instances which represents paths to files to be scanned.

Specified by:

getEntries in interface ParquetMetadataProvider

Returns:

list of ReadEntryWithPath instances whith file paths

getFileSet

public Set<org.apache.hadoop.fs.Path> getFileSet()

Description copied from interface: ParquetMetadataProvider

Returns list of file locations for table.

Specified by:

getFileSet in interface ParquetMetadataProvider

Returns:

list of file locations for table

getRowGroupsMeta

public List<RowGroupMetadata> getRowGroupsMeta()

Description copied from interface: ParquetMetadataProvider

Returns list of RowGroupMetadata instances which provides metadata for specific row group and its columns.

Specified by:

getRowGroupsMeta in interface ParquetMetadataProvider

Returns:

list of RowGroupMetadata instances

getLocations

public List<org.apache.hadoop.fs.Path> getLocations()

Description copied from interface: ParquetMetadataProvider

Returns list of file paths which belong to current table.

Specified by:

getLocations in interface ParquetMetadataProvider

Returns:

list of file paths

getRowGroupsMetadataMap

public com.google.common.collect.Multimap<org.apache.hadoop.fs.Path,RowGroupMetadata> getRowGroupsMetadataMap()

Description copied from interface: ParquetMetadataProvider

Returns multimap of RowGroupMetadata instances which provides metadata for specific row group and its columns mapped to their locations.

Specified by:

getRowGroupsMetadataMap in interface ParquetMetadataProvider

Returns:

multimap of RowGroupMetadata instances

checkMetadataVersion

public boolean checkMetadataVersion()

Specified by:

checkMetadataVersion in interface TableMetadataProvider

initInternal

protected abstract void initInternal()
                              throws IOException

Throws:

IOException