Skip to content

GH-3780: Decouple ParquetReadOptions from hadoop-mapreduce-client-core - #3781

Open
jerolba wants to merge 1 commit into
apache:masterfrom
jerolba:GH-3780_decouple_parquet_hadoop_from_hadoop_mapreduce_client_core
Open

GH-3780: Decouple ParquetReadOptions from hadoop-mapreduce-client-core#3781
jerolba wants to merge 1 commit into
apache:masterfrom
jerolba:GH-3780_decouple_parquet_hadoop_from_hadoop_mapreduce_client_core

Conversation

@jerolba

@jerolba jerolba commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Reading the Parquet read configuration via ParquetReadOptions previously referenced
ParquetInputFormat.getFilter(...) and statically imported its constants. ParquetInputFormat extends
org.apache.hadoop.mapreduce.lib.input.FileInputFormat, so using the read configuration forced the JVM
to initialize ParquetInputFormat and therefore FileInputFormat and its whole
org.apache.hadoop.mapreduce.* transitive dependency graph, even though only plain String/boolean
config properties were needed. The new ParquetInputProperties (constants) and ParquetInputFilters
(filter resolution) centralise the ParquetConfiguration-based read configuration so that read-only
consumers no longer transitively pull in the Hadoop mapreduce dependency.

What changes are included in this PR?

New files

  • parquet-hadoop/src/main/java/org/apache/parquet/conf/ParquetInputProperties.java
    • Declares all Parquet read-configuration constants previously on ParquetInputFormat:
      READ_SUPPORT_CLASS, UNBOUND_RECORD_FILTER, STRICT_TYPE_CHECKING, FILTER_PREDICATE,
      RECORD_FILTERING_ENABLED, STATS_FILTERING_ENABLED, DICTIONARY_FILTERING_ENABLED,
      COLUMN_INDEX_FILTERING_ENABLED, PAGE_VERIFY_CHECKSUM_ENABLED, BLOOM_FILTERING_ENABLED,
      OFF_HEAP_DECRYPT_BUFFER_ENABLED, HADOOP_VECTORED_IO_ENABLED, HADOOP_VECTORED_IO_DEFAULT.
    • Constants only; no org.apache.hadoop.mapreduce dependency.
  • parquet-hadoop/src/main/java/org/apache/parquet/conf/ParquetInputFilters.java
    • ParquetConfiguration-based filter resolution: getFilter(ParquetConfiguration),
      getUnboundRecordFilter(ParquetConfiguration) and getFilterPredicate(ParquetConfiguration).
    • No org.apache.hadoop.mapreduce dependency.

Removed

  • parquet-hadoop/src/main/java/org/apache/parquet/ParquetInputConfiguration.java (intermediate
    class, split into the two classes above).

Modified

  • parquet-hadoop/.../ParquetReadOptions.java
    • Static imports now resolve to ParquetInputProperties, and the getFilter(...) call to
      ParquetInputFilters instead of ParquetInputFormat. ParquetReadOptions no longer references
      ParquetInputFormat.
  • parquet-hadoop/.../hadoop/InternalParquetRecordReader.java
    • Static imports of RECORD_FILTERING_ENABLED / STRICT_TYPE_CHECKING now from
      ParquetInputProperties.
  • parquet-hadoop/.../hadoop/ParquetInputFormat.java
    • Constants are now @Deprecated, delegating to ParquetInputProperties.*.
    • Kept the legacy Hadoop Configuration-based getFilter(Configuration) /
      getUnboundRecordFilter(Configuration) overloads (used by the MapReduce read path), delegating by
      wrapping into HadoopParquetConfiguration. Internal callers use ParquetInputFilters /
      ParquetInputProperties directly.
  • Tests updated accordingly (DeprecatedInputFormatTest, TestParquetFileWriter,
    TestInputOutputFormat, TestInputFormatColumnProjection, TestDataPageChecksums,
    TestColumnChunkPageWriteStore, TestPropertiesDrivenEncryption).

Are these changes tested?

No change in behavior, just refactoring code location. Code compilation and existing tests validate the change.

Are there any user-facing changes?

No. Fully backward and binary compatible: the ParquetInputFormat constants and methods are retained
(deprecated) and delegate to the new class; constant string/boolean values are unchanged, so any
previously-serialised configuration still works.

Closes #3780

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decouple parquet-hadoop module from hadoop-mapreduce-client-core

1 participant